I had the same problem with Javascript so I used I used Selenium to drive Chrome to take a screencap of the page. Then I used K-Means clustering with EM to convert the pages to their constituent colors.
I scraped 100 and 1000 of the Alexa top 1M. Cool to see another approach, great work!
"It’s easy to notice a bug when examining the colors for Google (note, this is normal google.com not a doodle). Notice how the three colors are light gray, dark gray, and white – not the typical red, green, blue, yellow color scheme. Why? Well, when the image screenshot is resized to 320 x 240 pixels for processing, the colors are dithered. The number of pixels in the new image that lie between red, green, blue, yellow and white – the dominant background color – is much larger than the number of pixels that are colored. Because of dithering, those between pixels are closer to shades of gray, than colors, and thus the k-means clustering (with EM) finds shades of gray and white to be the “color of Google”. I’m not sure if this is a bug.. what do you think?"
That's awesome! I figured someone else must have had the same idea before me. :)
I think your screenshot scraping technique is probably more accurate than my text parsing. I also like that you used a larger sample size. I plan to experiment with groups of 100 and 1000.
Thanks for sharing! It's always interesting to see how different people achieve similar goals.
I'd like to begin scraping the images on the sites soon too. When I've got a good chunk of time I'll look through your source code for inspiration. Mind if I reach out with questions when I do?
EDIT: I also really enjoy those woodblock prints! Now I want to somehow print my data for the top ten sites onto canvas.
Rather than resizing to 320x240, pick that number of pixels randomly. For even better results use some method of variance reduction e.g. divide the screen into n squarish rectangles and pick N/n pixels from every rectangle.
I'd second using selenium / webdriver for this. You wouldn't need a great deal of code to get the raw pixel data.
The area I suspect will need thought is how one gives "value" to distinct colours. It's tempting the go by the amount of screen space taken up, but that ignores the way the eye and mind works - a red dash in a sea of another colour needs to respect the presence of the red as it could be critical. But then there are sections where the colours might be low in importance (eg colour of text in legal text from a footer)
Yeah, that seems like a great option. I'm not sure how to get around the dithering bug that dynode ran into though. I'll have to do some research before I get started on V2
It's not that bad.. typically you use a 32.768kHz ("watch") crystal then divide by 15 to get one second. You can do it with two chips (a bunch of solutions, one is two 7468).
To nit-pick, of course the division is not by 15 (that'd give 2,184.5333... Hz) but by 2 to the power of 15, i.e. 32,768. Typically implemented by a 15-bit counter.
You need 15 triggers to go from 32768 Hz to 1 Hz. Existing circuit uses 4 triggers to divide 60 Hz by 10 and another 3 to divide by 6. You can repurpose those so you only need 8 extra triggers. Two transistors each means 16.
You can also save 6 transistors from existing dividers because all divisions are by 2. So net total is 10 extra transistors for division, plus at least one in the oscillator itself.
First thing to do is have a look at the datasheet <http://www.eonssi.com/upfile/p200892917341.pdf>. I was hoping that it'd be SPI or I2C, but it's a big parallel flash, so reading it is going to take some hardware.
Comes in two packages a TSOP and a BGA. I'm hoping you have the TSOP. If you've got a TSOP, you can get a socket for it / probe it. You can probably also get a socket for the BGA but you might be paying out the nose.
I like the JTAG approach as well. Assuming it's connected to a processor or FPGA that's got boundary scan, even if the rest of the system isn't cooperative you can bitbang the contents of the flash out using the JTAG hardware.
The JTAG silicon - even if it's on the same die as the processor / FPGA is totally separate. It's something like a tiny state machine. Read up on it, really powerful. <https://en.wikipedia.org/wiki/Boundary_scan>
Another option is to transplant the interesting flash from the dead device to an identical working device. Use the working device to read out the contents of the memory. This will require some soldering skills.
Some more info - openocd is a not bad open source JTAG server. <http://openocd.org/>
The TL866 I suggested does support your part.
If you've got the flash in a TSOP - another trick for getting at the flash is to find a FPC cable that has the same pitch as your TSOP and solder it down on the board. FPCs with varying pitches are available at Digikey etc. That way you don't have to remove the flash from the board.
An integrated oil company like Exxon has activities that lose money when the price of oil goes down (drilling, exploitation) and others that make more (refining, retail). The price of oil dropped by about half during the past few months; Exxon barely budged.
A better proxy for oil price could be drillers (Diamond Offshore Drilling for example) but they're very far from perfect either. In general, it is very hard to track an asset which price is expected to be mean-reverting like the price of oil.
I use the multiprocessing module in python all the time for quick parallelism <https://medium.com/@thechriskiehl/parallelism-in-one-line-40... Let's you easily map to multiple cores. I use it a lot for image processing tasks. Quickly crawl through directories to find all the files, then spin up all the cores on my machine to crunch through them. Wish there was an easy way to enlist multiple machines.
I too remember something like that, but was under the impression that CAs are still ok.
But of course, judging by the massive downvoting you've gotten, I suppose you're incorrect. I wish those downvoters would explain their viewpoint rather than downvoting...
I agree 50/50 split is best, I agree with most of this [1] article (recently posted on HN) about founder equity.
I also agree that choosing non-profit will most likely help our cause, it would take the company in a different direction. This is something we'd prefer to discuss with smarter folks (hence the desire to delay some of these decisions).
I'll keep dissenting: 50/50 is usually not the right solution. Please google founder equity calculator, and use it to start a conversation about who does what and who is committed.
For the YC application, just say that you are not incorporated yet and that you are thinking of splitting 50/50, that should be good enough for them to review.
We used a G1 dev phone to fly a 1.5 meter wingspan airplane. We used only sensors in the phone, camera, gps, etc.
Honestly it didn't work too well - we needed either a much better model of our airframe, an airspeed (not groundspeed) sensor or an airframe that could power through 10-15 mph winds.
In the end however, we demoed waypoint navigation and imaging the waypoint. You text it lat/long and it texts back a picture.
We did it as a proof of concept (for an interested govt agency), modern smartphones are neat, but bespoke, real-time hw, with navigation grade sensors are probably an order of magnitude more effective (though an order of magnitude more expensive).
Working with android to do robotics was.. interesting. We had to do all sorts of hacking to make the phone behave a little as possible like a phone and more like a real-time controller. What do you do when someone calls the phone while you're flying an airplane? How do you keep the phone application from taking over and exiting out of your navigation app? While the airplane is 1000 ft in the air?
What I ended up using for the ground/water stuff was sending serial commands out of the headphone jack... surprisingly low latency, especially compared to bluetooth, and it was only a few bytes at a time so low baudrate was fine.
