Programming

disclaimer: I work primarily with SQL Server and have almost no direct experience with MySQL, but…

In general I think temp tables are a legit approach to the sort of reporting scenario you describe. For particularly gnarly queries, they can help with readability, if nothing else, which is sometimes worth it all by itself.

It sounds like you’ve already picked off the low hanging fruit (nested loops in the application, etc.) and if you’re seeing a 10x improvement in execution time, I’d call that a victory. It’s not uncommon to see variations in performance of the same query for different volumes of data, or even similar volumes with different distributions of values. That’s what makes performance tuning so much fun! :cry:

Another thing to consider is that some times you can only do so much with the query itself. If the database was poorly designed – and I’ll admit statements like this

make me think that’s a distinct possibility – you might have bigger problems than can be solved by rewriting queries.

Even so, there are things you can do w/out altering the schema. For example, it might be worthwhile to look at your indexes and statistics; sometimes adding the right index can make a huge difference for a particular workload without even changing your queries. Again, I don’t know specifics for MySQL, but I can’t imagine the core concepts are that different from what I know.

Finally, if you really wanted to do instantiated denormalized tables, I’d probably be tempted to just go all the way and build a datamart or data warehouse or whatever they’re calling it these days for your platform of choice. But that’s whole different scenario, with it’s own set of trade-offs, and likely a fair bit more work than was envisioned when your boss asked you to fix these reports.

Or you could just throw hardware at the problem. :grin:
(seriously, though, there’s no shame in a tactical approach to tackling scaling issues.)

7 Likes

Oh yeah, we’re partway through a multiyear project to re-architect and rewrite everything. But we have to keep the current version working (and avoid losing customers!) while we do that. Also learn from the last decade’s work so that we don’t make the same mistakes.

Definitely found some of those cases! Also found a case where we have lots of duplicate indexes on a table; looks like someone else was trying to figure out a way to fix something and having trouble solving it with indexes. The temp tables (or instantiated ones) are for where indexes fail me - joins via variable-depth hierarchies of variable type and cardinality. Even with indexes, those have a really tough time.

I’ve brought that up a few times as something to look into for the new version. It would be a major change into territory that none of us are familiar with, and we’d have to change some expectations, but I think in the long term we’re going to need to try out something like that. Not a candidate to get retrofit into the current version though.

Oh we’re definitely doing that too. :joy: I’m pushing hard for more RAM because we’re wasting so much redoing the same processing over and over, where with a bit more RAM we could cache and skip a lot of that. But of course it’s a bit more RAM per instance multiplied by hundreds of instances which adds up quickly on the bill. And we’re pushing other boundaries too, so that’s a juggling act.

But as problems go, having too much business and growing rapidly is a good one to have.

6 Likes

:point_up: :sunglasses:

9 Likes

As someone that’s written several SDKs this was filled with facepalm worthy moments.

I’ve used this API before. From a producer context, it’s really good (especially these days) - you can really easily set up powerful and fast structured tracing with it. From the consumer perspective that this article refers to, it can be bloody awful.

5 Likes

I was debating putting this in the games thread or here.

7 Likes

Great. When do we get the rest of the Apogee family?

4 Likes

Not sure if this is the right place to put this, as I’m not really asking for a solution. GMail (or Firefox, or Windows, or the video driver, or the monitor) is doing something weird with smoothing letters:


I’m vaguely aware of the color difference (e.g., between line 1 and 2) even when the letters are only 25% this size, and it’s really annoying. Is this typical?

4 Likes

Are you running Windows 10? It does have an anti-alias option built in (might be in earlier versions as well, but that was the first time I had to use it).

4 Likes

Yeah, Win 10. But I don’t see any difference with the antialiasing turned on or off. And it seems to vary between programs – here’s Libreoffice Writer on the left, and Notepad on the left. More or less the same text size (12).

4 Likes

I did have some apps run without the anti-aliasing, even though it was turned on. It depended on if they were running in a compatibility mode or not. I forget which way the settings went.

Let us know the root cause in your case.

4 Likes

Have you tried cranking up the resolution?

4 Likes

This might proive useful

6 Likes

I’ll have to check that out. I’ve had some programs that looked really blurry as Windows attempted to make them look better; I have that turned off in a lot of cases. I’ll let you know if I find anything.

The resolution is set the same as the max on my monitor, 1920 x 1200. I have tried fiddling with the monitor controls, but “other controls” (other than brightness and contrast) is grayed out for some reason.

@jerwin, thanks for the tip on ClearType! I’ll have a look at that.

4 Likes

Tuning ClearType seems to have helped. Things seems more consistent.

6 Likes

https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers

Not that there’s anything wrong with $9/hr programmers… for writing a web site or something. Probably not the best idea for something critical to life safety.

11 Likes

People wonder why tribal knowledge is a thing. An indispensable thing.

8 Likes

I’d love to see:

boeing executives health care outsourced to 9 an hour doctors

10 Likes

Hi everybody!

11 Likes

Hi dr Nick!

7 Likes
11 Likes