I was in the US and want to immigrate to Canada because of US broken H1B immigration system ( for me it will take 9 years to get a greencard). Here is what I did -
The immigration website of Saskatchewan province opens up randomly to apply for immigration. I missed it many times because there is no indication other than the "Apply" button being enabled and a small text in their homepage which says "Applications are now 'open' ". They will close the application intake when they have reached X number of applicants. So timing is very important.
So I hacked up a script which diffs their home page every 10 minutes for "open" regex. When there is an "open" keyword in the diff, the python script calls twilio API to make a phone call to me along with an SMS.
So this script was running in AWS for many weeks and one day I got the call. Logged in to Saskatchewan's immigration homepage and applied. Now I am in Canada as a permanent resident. Thanks to Twilio.
edit: add H1B to make clear which type of immigration is broken IMO.
(This comment is from my previous HN account, for which I lost my pwd.)
If you want to build stuff like this in future, check Page.REST a simple API I built. You can pass a query string `contains=TEXT` to check the existence of a word or a phrase.
I was hired to scrape some economic related pages and build an excel file and email it, I got that done but not sure if I should try to host this and turn it into a service or just set it up for the client and let them deal with it. It's just personal use on their part.
I would not equate scraping with periodic searching for a keyword.
The problematic part is when you scrape data off of websites, and the owners don't want you to do it; as in, they would not even be happy if you manually copied that stuff into an excel file for fun or profit.
My counter to this is, without Google's "algorithm" the search results would not be great/find what you want. And in this case websites want to volunteer/agree to index their site for visibility but then you still have to visit the site because conveniently The site summary ends before the content that you want. I don't know how I would find websites unless referred on some website.
I kinda felt the same way about how Netflix started out but I think they probably had a deal to pay a portion of revenue from dvd's that they rented out.
Because this was very important to me and my life forward, I thought it was better to roll my own custom notifier. Moreover, the phone call thing was the most important part which will get attention any time of the day than an email.
Wait a while...it took about six months for one to happen here in Sydney. Well worth the wait - half an hour of exclusive use of a Hololens. All the while being recorded for market research purposes, but that seemed like a fair trade.
Did the same thing to get a wedding registration appointment with the Berlin city administration, but used Ghostery instead of AWS+Twilio. The responsible administration department currently doesn’t give out appointments and advises not to plan weddings because they are so understaffed, but I didn’t feel like waiting.
Got a notification email from Gostery in the middle of the night, registered the only available slot et voilà!
I think it is a passing thing. Germany recently legalized Gay marriage and presumably that lead to a huge backlog of gay people finally wanting to marry.
So we both had life changing events. Its the real winning moment of a programmer when he is able to use his/her skills for a "non-work" task automation. Its more satisfying.
I did a very similar thing to check if my H1B was approved. My use case was simpler - I regexp matched a keyword on the webpage and converted the script to an alexa skill deployed locally.
Unfortunately, i wasn't home for the week when the approval happened :P. But it was fun asking alexa to check the status!
Did something similar to check availability of a special deal by my ISP.
No need to write code, you only need a shell account on a box somewhere. Put a line like this in your crontab:
20 9,12,15,18,21 * * * curl -s http://example.com/special-deal-page | grep 'something that needs to appear on the page' | mailx -E -s "it's happening!1!" $MAILTO
This will check every three hours during daytime (which is what I wanted).
"mailx -E" will only send an email if the input is not empty. Don't forget to define MAILTO and to test the contraption.
I used this same technique to apply for an early interview for the U.S. CBP's Global Entry program. Someone already open-sourced the tool, then I set up a cron job to run it: https://github.com/davidofwatkins/ge-cancellation-checker
Yes, made it to SK, but moved to Montreal for better job prospects. Probably will have to move to Toronto because my spouse's can't find a good job due to the strict French language requirements for non IT jobs in Quebec.
Frankly, I didn't think much. Just wrote this script in an impulse of a moment when I missed a slot. During that time, I didn't think about how it was done, but the end result. Sure diff would have triggered false positives, luckily didn't get any.
To be clear, H1B isn’t an immigrant visa. It’s a temporary skilled worker visa. There are other visas designed for skilled immigrants. Someone on an H1B isn’t an immigrant so of course the system would be “broken” when expecting H1B to result in a green card. It’s no different than expecting a tourist visa to be the first step to getting a residency visa. H1B was never implemented as a pathway to permanent residency.
Still that aside, great ‘hack’ and congrats on getting your Canadian residency!
Thats wrong information. H1B is a “dual intent” visa.
Its broken because it was really intended for exceptionally skilled workers. But currently its a lottery. USCIS just takes the word of the company applying for it and grants the visa. This loophole is being exploited by numerous body shops. That is reason why there are low quality H1B workers. There are various other problems like employee mobility, spouses of H1B is not free to work in the land of the free. Country of birth based quotas for permanent residency, rather than skills or points based quotas.
You could write this with simpledomhtlm + php + cron or a raspberry pi (cron) and then send yourself a webhook notification via slack or configure mailgun (mailing client) or firebase android notification (haven't done this).
Inspect the page,find the structure, isolate or if it is easier to just dump the entire plaintext and use strpos to find a specific word.
Yeah, was about to say, it's too small of a niche. Plus the risk that sooner or later, Saskatchewan would update their site to provide the same feature.
Yes, too small of an audience to sell it to. Also I didn't care about the timezones in the script. It will make the call no matter in which timezone you are in (you will get the call in the middle of the night), because Saskatchewan's site do not care, it will close when the quota gets filled and it gets filled fast, within hours !
Well, you'd want to make it a /bit/ more generic, surely? Like, "load this url and send a text to this number if the (page text | diff) matches this regexp"?
Even allow people to create presets that would then be added as a generic thing. You could use it for camping all sorts of intermittent online things (web site specials, online ticket pre-orders, etc.)
There are many sites which already does this. During that situation when I missed a slot, I didn't want to take any chances again by using sites like visualping. I trusted my script and twilio API to work.
First I thought to ring a loud alarm connected with a Pi, but then it will be limited to my apartment. So I went with twilio to give me a phone call. Because I should act immediately and time was of the essence. Emails or SMS might go unnoticed.
The problem with monetizing this service is because there are many immigration watsapp groups with many members in it. So when one member buys this service, he can simply broadcast this notification to his watsapp groups. I even made the service free, but no takers. Anyways it worked for me, changed my life !
I get what you mean, but the same might be said for someone being able to check manually every 10 minutes. Then there is no equal opportunity between the working and workless classes.
The immigration website of Saskatchewan province opens up randomly to apply for immigration. I missed it many times because there is no indication other than the "Apply" button being enabled and a small text in their homepage which says "Applications are now 'open' ". They will close the application intake when they have reached X number of applicants. So timing is very important.
So I hacked up a script which diffs their home page every 10 minutes for "open" regex. When there is an "open" keyword in the diff, the python script calls twilio API to make a phone call to me along with an SMS.
So this script was running in AWS for many weeks and one day I got the call. Logged in to Saskatchewan's immigration homepage and applied. Now I am in Canada as a permanent resident. Thanks to Twilio. edit: add H1B to make clear which type of immigration is broken IMO.
(This comment is from my previous HN account, for which I lost my pwd.)