I really love the wave of new tools that are gaining traction in the Python world: tqdm, rich (and soon textual), fastpi, typer, pydantic, shiv, toga, doit, diskcache...
With the better error messages in 3.10 and 11, plus the focus on speed, it's a fantastic era for the language and it's a ton of fun.
I didn't expect to find back the feeling of "too-good-to-be-true" I had when starting with 2.4, and yet.
In fact, being a dev is kinda awesome right now, no matter where you look. JS and PHP are getting more ergonomics, you get a load of new low level languages to sharpen your hardware, Java is modern now, C# runs on Unix, the Rust and Go communities are busy with shipping fantastic tools (ripgrep, fdfind, docker, cue, etc), windows has a decent terminal and WSL, my mother is actually using Linux and Apple came up with the M1. IDEs and browsers are incredible, and they do eat a lot of space, but I have a 32 Go RAM + 1TO SSD laptop that's as slim as a sheet of paper.
Not to mention there is a lot of money to be made.
I know it's trendy right now to say everything is bad in IT, but I disagree, overall, we have it SO good.
New features get added and these posts are a nice reminder to check the docs on a tool I literally never need to do that for because it's so habitual to wrap any long-running iterable inside of a `tqdm()`.
Yeah, I've been using it for a while, but with the barrage of memes and politics on the internet nobody finds out about the good stuff until years later.
Way before @noamraph put it up on in pypi it was internally available in our company. To my knowledge he didn't yet make it 30 years ago, but if he did then probably at least not for Python!
Jokes aside it's a great language. I would love to see a better package management ecosystem for it, as that is also my biggest issue. Python also does something no other mainstream scripted language does- it allows you to install extensions to the language right in the same package manager, as it can compile libraries from source when wheels are unavailable- this makes it a much harder challenge. At the same time I'm really happy that PyPI is a non-profit organization and won't have to go through the issues that something like NPM did.
I love Python (flexibility, ecosystem, datamodel). I love Java (verbosity, explicitness, robustness). Now, kill me. I don't let others define what I should and shouldn't think about a tool, in this case a programming language. I make up my own mind based on my own experience with the tools. Remember, they're just tools. Not the end game.
I really like Poetry for my own uses but as a semi-infrequent Python user my struggle is dealing with other people's Python repositories that aren't using it.
I can never remember all the differences and subtleties between virtualenv, pipenv, venv, pyenv-virtualenv, workon, conda, and so on when I encounter them in a random git repo.
I did AoC 2021 through Day 7 in Python, before switching to Go (for personal learning, can't remember if times listed below were with "go run" or from running after "go build"). D7 actually in both, to try and solve a problem I was having with my Python performance. But, it was actually a big learning moment for me when it came to structuring my logic.
My Python code in Part 2 (some minor adjustments from Part 1) took about 40 seconds to run. Terrible, but usable to get a proper answer. I was able to bring it down to ~25 seconds with some optimization by adding a calculated lookup dictionary per loop. Now, the same exact logic in Go ran in about 0.8 seconds.
However, I wasn't satisfied with this and realized that if I moved the dictionary to be global rather than per-loop, I would be able to realize significant gains in performance by completing eliminating redundant calculations. This change dropped the runtime from 25 seconds to 0.35 seconds (of course, applying the same logic in Go brought it down from 0.8s to 0.05s).
Due to the nature of performance you can get out of the Python interpreter, it can actually lead you down paths of learning better optimization strategies that you may initially write off in other languages (depending on the use case) because they perform inherently better. It made me think a bit more about what I was doing and how I could improve it since (in this particular case), the impact of not doing so was pretty drastic.
Day 6 and 7 are fantastic for that. It's a learning moment, where you can get the benefit of thinking about the problem for a few minutes instead of bruteforcing gives you a dramatic speed up.
When I read the Day 7, I saw bruteforcing would lead to bad perfs. Then I remembered that in high school, I learned a formula to calculate the nth term of a sequence without having to process the entire sequence.
I couldn't remember the formula, nor the name of the concept, so I google around until I found some tutorials, and relearned what I was taught as a child: arithmetic sums.
The consumption for a crab can then be calculated in constant time:
Interesting, I'll have to look at that. Actual algorithms are a big weak point for me, I'm not a developer by day, so I don't spend a lot of time learning code practices or computer science/math topics. This was my solution to Day 7 Part 2 (positions = sorted int list of input data):
def calculate_fuel(positions):
fuel = None
low_fuel_value = None
calculated = dict()
for value in range(positions[0], positions[-1]+1):
consumed = 0
for position in positions:
diff = abs(value - position)
try:
consumed += calculated[diff]
except KeyError:
if position != value:
consumption = sum([x for x in range(1,diff+1)])
calculated[diff] = consumption
consumed += consumption
if fuel:
if consumed < fuel:
fuel = consumed
low_fuel_value = value
else:
fuel = consumed
low_fuel_value = value
print(f"Aligning to: {low_fuel_value}")
return fuel
And Day 6 I fell for the bruteforce bait and had the thought "It can't be as easy as changing 80 to 256, right?". Then I realized the pain I had created for myself. BUT! My 6p2 code ran faster than my 6p1 by a good margin, which I was happy about.
Can even calculate the target without bruteforcing. . n^2+n is approximated by n^2 when n is large. Can take the mean of all the distances and use that (or perhaps +/- 1).
And similarly for part1 take the median. Why it kinda works:
Part1: The median I felt made sense intuitively, as in my head I thought about an example ala 1,1,3,100. Never makes sense to use x>3, because even though the crab at x=100 then can walk shorter, there are 3 others then having to walk longer. And x=1,2or3 doesn't matter, just symmetrically changes which side has to walk one step less or one step more.
And for part2 I thought similar, except the cost is exponential and therefore I want to minimize the avg move and not the total moves, thus taking the average.
The better optimization with Python is to not use python. Half joking. If you want awesome performances, transform your problem into a numerical one, and use Numpy. Numpy is awesome. I always miss it when using other languages than Python.
I don't understand your solution to Day7. What kind of dictionary, and how come even a naive solution would be so slow?
My Kotlin solution runs in about a second. And it was even so stupid that I didn't calculate the sum of the arithmetic series directly, but through a loop. Can't fathom something being slower.
line.split(",").map { it.toInt() }.let { crabs ->
(0..crabs.maxOf { it }).minOf { pos ->
crabs.sumOf {
(0..abs(it - pos)).sum() // slower than calculating arithmetic sum, but quicker to write
}
}
}
My proper Kotlin solution runs in less than a ms, though.
I do think that "it's good that python is slow because it forces you to optimize" is a weird take, though.
> I do think that "it's good that python is slow because it forces you to optimize" is a weird take, though.
My take wasn't that it forces you to, but that non-optimal code paths can be greatly exaggerated in comparison to other languages, particularly compiled ones. You can still ignore it (I mean, within reason), but it can give you that extra push to really look a bit deeper to understand what's going on. And of course, there's optimized libraries written in C/C++ that you can take advantage of for even better number crunching than standard CPython.
> What kind of dictionary, and how come even a naive solution would be so slow?
My naive solution was literally going through every single element for every loop and not storing any data besides the fuel buildup and the alignment number that generated it. The dictionary was added to act as a cache to store already computed fuel consumption values, initially per-loop then moved one level up to be global (because the summations wouldn't be different).
I'm not saying my method (posted in a sibling comment) is the best solution, but it's the way my brain walked through the problem.
Posted my optimized Kotlin solution in that sibling thread :)
Cool of you to participate without being a developer! Lots of computer science topics makes it easier, so hard without knowing of them. For instance graph searching / Dijkstra has beem relevant this week.
Yeah, I was doing a CS minor in college but had to drop as it was consuming too much of my time from my other discipline in the non-intro courses. Big(O)/time complexity were my usual failings in the intro algorithms course I took.
I'm not unfamiliar with programming, but I come from the sysadmin side of things. "Glue" work is usually where things are focused and the 'fun' nitty-gritty of algorithms can be a bit out-of-scope, though I'm not a sysadmin in my current role anymore so any dev-related work I do is purely personal now.
I've had to take a break from AoC, only got up through Day 10, but didn't get P2 for 8 and 9. It's a fun way to keep the mind going and to slip back into the coding space to at least not lose skills, even if the solutions are simple/non-optimal.
Yes! Tasks like AoC (relatively small, without stringent performance requirements but still requiring the correct algorithm) are where Python is not only a reasonable choice, it's unreasonably effective.
Doubly so this year, with the theme being linear algebra.
doit [0] is a superb toolkit for building task-oriented tools with dependencies. It isn't too complex to get started and gives you a lot to work with. I've used it for an internal tool used by everyone in the company for 5+ years and it has never given me a headache.
I mean, it's declarative, works on Windows, easy things are (very) easy, and because you can mix and match bash and python actions, hard things are suspiciously easy too.
Given how complicated the alternatives are (maeven, ninja, make, gulp...), you'd think it would have taken the world for years.
Yet I've only started to see people in the Python core dev team use it this year. It's only getting traction now.
Here is a task that groups all static files from a web project into a directory.
It has to make sure a bunch of directories exist, run a node js command to build some files from the proper dir, then run a python command to regroup them + all static files from all django apps into one dir. Simple.
But then I had a problem I had to hack around. This required me to change an entry to a generated TOML file on the fly at every build.
doit just lets me add a 5 lines python function that does whatever I want, and insert it between my bash tasks, and I'm done.
It is a pain to work with invoke now. It is all find and dandy for the basic features but you going to hit the seams soon you start trying advanced stuff. Looks like the project is going to be abandoned.
Love the idea of Schema, but not a fan of that syntax, doesn't seem Pythonic to me, seems more like something you'd see out of Javascript word.
Like looking at that very first example, I have no clue what "len" means in that context. Is it implicitly checking that it's not an empty string? Then on the next line, how come `int` has `Use()` around it, but on the previous line `str` didn't? I guess that int is being used as a converter, line on the next line with str.lower, but the str was being used as a type check?
I'll add BeeWare (https://beeware.org/) as a pretty nice nice to deploy Python applications cross-platform easily. It's more like a suite of tools, though.
I love Python for many things: scripts, data science, prototypes, etc. But I would never use it to build a large backend system. I just can't handle all the warts. Yes it has been used by large companies, but not without their problems.
You use Python to rapidly build a small backend system and then, if any warts appear that need to be smoothed over, you carve that function off to a more robust solution.
Python is great for programming at the speed of inspiration.
for prototypes and MVPs time-to-learning and time-to-market are king, and programmer time is the most expensive thing. in that niche Python is helpful and tests just slow you down
tests are super helpful and wise in any sort of long term or safety-critical software, obviously
so, tests (and Python) are not inherently a win or a loss. they just give you different trade-offs
times to write testless Python and times to write test-heavy Go, Rust etc.
Generally I believe in engineering tradeoffs but I think some tests are always important. Even with MVPs. Writing test might double your time to market, but they will allow you to move faster and make less regressions once in production. The early days are some of the least stable features wise and having tests in place to verify things when massively changing a codebase is nice. Also, it can greatly help add confidence to newcomers who don't know the codebase.
understood. theres def an art to judging when its too early to write tests vs the perfect time. its easy for anyone to recognize once its become "too late" and therefore more painful and costly to add it than otherwise would have been
I therefore try to put each situation quickly into 1 of 3 buckets then move forward on that basis:
1. heck no
2. heck yes
3. either way. a gray area
I generally see a case 1 and 2 with confidence. therefore by a process of elimination, that also lets me deduce when its case 3. and in those cases you cant go wrong. :)
I think that is also reasonable, if the production system does not have to service a heavy load, and does not need to be scalable. Scalability is often assumed to be a requirement, however it is not always required in practice.
It's energy inefficient like a school bus, not inefficient for the sake of it, but because it's a decent way to shuttle passengers of all experience levels through all weather.
Do you seriously think that 20 cars are more economic than a single bus?
Python performance problems are exaggerated for many use-cases (hot paths are not written in Python e.g., matching regexes happens in C code(re,regex), parsing xml too (elementtree, lxml), sqlite, numpy,scipy (C, Fortran), etc. Cython makes it trivial to drop to C level where necessary)
With the better error messages in 3.10 and 11, plus the focus on speed, it's a fantastic era for the language and it's a ton of fun.
I didn't expect to find back the feeling of "too-good-to-be-true" I had when starting with 2.4, and yet.
In fact, being a dev is kinda awesome right now, no matter where you look. JS and PHP are getting more ergonomics, you get a load of new low level languages to sharpen your hardware, Java is modern now, C# runs on Unix, the Rust and Go communities are busy with shipping fantastic tools (ripgrep, fdfind, docker, cue, etc), windows has a decent terminal and WSL, my mother is actually using Linux and Apple came up with the M1. IDEs and browsers are incredible, and they do eat a lot of space, but I have a 32 Go RAM + 1TO SSD laptop that's as slim as a sheet of paper.
Not to mention there is a lot of money to be made.
I know it's trendy right now to say everything is bad in IT, but I disagree, overall, we have it SO good.