Monkey see, monkey love

Disclaimer: The following content expresses my own opinion and does not aim to represent circumstantiated facts.
Google, IDEA, IntelliJ, Java, Kotlin, and other names, brands, products mentioned in this post are trademarks of their respective owners.

Everybody loves Kotlin, right?

It’s the best language on the JVM! It’s as fast as Java, less verbose, it provides better null handling (because heaven forbid that we stop using them), etc, etc… A true miracle of modern software engineering!

And it’s also young, cool and if you are not using it well… you are obviously a moron!

“Why?” you ask? Well, because everyone says so, of course!

And why does everybody say so?

To give my answer to this question, let’s go back a couple of years…

True Story!

I happen to work in a great environment, yet many of my coworkers – while being good fellows – are of the most conservative and rigid sort.

They have Java (and Typescript which I think I can safely call “browser java”…), Java is their shepherd and they shall not want.

When I tried to interest them in anything else, the reaction has always been: “back off, Riccardo, with all your Clojure and LISP nonsense!”.

But then they added:

“Kotlin seems interesting, though…”.

“Why?” I ask.

No conclusive answer.

They know nothing about it. They never read a book or an article about it, they never coded with it, there is not a single project they know about where Kotlin is the key factor. They just happen to “find it interesting”.

And let’s be clear, most of them never heard of Lua, Clojure, Elixir, Erlang, Haskell, or other languages before knowing me. Maybe (just maybe) they heard about Scala.

But still, they do recognize Kotlin as a potential Java replacement. They know it’s most likely a good thing.

This leads us back to the first question:

Why it seems that anybody considers Kotlin the best next thing after sex?

Conspiracy time!

Here is my little conspiracy theory: keep in mind that everything that follows is pure speculation, I’m not pinning anything on anybody.

Kotlin is an IntelliJ creation, the same company that makes the (excellent) Idea IDE: that they are pushing their language on every turn is no surprise.

But then, there is also Google.

Google is all over Kotlin, with its tech evangelists oozing loads of virtual ink on it, Bukkake-style.

Did you know that Kotlin is the best language for Android development?

I’m sure Google would have let you know by now.

Care to try a new young, cool, slim language? Kotlin is the language for you: even we, at Google, use it!

How many Google developers are actually using Kotlin by the way? 50%? 107? 2? We will never know…

It would look as if Google is drinking the Kool-Aid from IntelliJ, except that Google is smart and we – the developers – are the morons who are drinking it!

I don’t know if this is really how it shook out, but If I had to put my money somewhere, I would bet that Google received Android Studio from IntelliJ almost for free.

Why?

Because Google used us suckers as a currency to pay them.

How?

By pushing for every technology Idea puts into Google’s mouth.

Kotlin could be the best language for Android development because it’s the language provided to Google by Idea and because – I would think – Idea has also provided all the manpower for the porting Kotlin on the platform, just as I’d say they provided the manpower to put a bow on IDEA and gift it to Google as Android Studio.

Google is pervasive and ubiquitous: should Idea want to shove Kotlin down our throat, would it be so strange for it to want to have Google on its side?

Remember when I said that “it seems that anybody considers Kotlin the best thing after sex?”. Well, I think that the “anybody” I was talking about is mostly Google.

The apple of our eyes

Take the most rigid, focused, and conservative Java developer on the planet, and, no matter how indifferent he may be about other languages, you can bet IDEA or Google has slapped Kotlin on his face at least once a day.

And, I think, this is all it takes to inspire trust in otherwise rational human beings.

“If IntelliJ created IDEA which is so damn good, then Kotlin must be good as well, right?!”

“If Google that’s so big and powerful uses Kotlin, it must be that Google is big and powerful because it uses Kotlin!”

It’s as I wrote in the title: “monkey see, monkey love”.

I think I can almost hear you.

“How do you dare to compare us to monkeys, Sir! We, the developers, are not common human beings! We talk with computers! We have degrees in Physics, Mathematics, Engineering!”

This is the hubris that gets us: we think we are so special, so rational, so smart, but in the end, as humans, we have just the same cognitive biases as everybody else.

Conclusions

“But Riccardo” you may say “even if that may be true, in this way Google doesn’t have to develop a half-assed IDE, Idea gets the advertising it deserves and we – the developers – get great tools to work with. Aren’t we all winning? What’s the problem?“.

The problem is that the same company that created our IDE may be aggressively pushing for its (mediocre) language to replace the one we came to it for in the first place, and could be doing it without being upfront about it.

Another problem is that it may be trying to establish a monopoly and incorporate us into an ecosystem where they control all the variables.

Isn’t it funny?! We got to them because they provide the best IDE for Java around, and they could end up using it to defeat the competition and making us leave Java in the first place!

Doesn’t that bother you?

Well, maybe you are right. I’m overreacting. Don’t worry about it.

Have a glass of Kool-Aid, instead: it’s on me!

Small dictionary of moronic app reviews

Let’s rant, for a change!

Years ago, I published a small VR video player on the Google Play Store, called VR Theater.

VR Theater is admittedly not a great app, but it has something going on for itself: it’s completely free.

Not “free with ads.”

Not “free, but I will sell your personal data.”

Not “free for a limited time or “free with in-app transactions.”

Just free.

That doesn’t mean I got nothing out of it: visibility and plenty of great reviews from outstanding users, but I don’t charge anything.

Not all reviews are great, though, and while I don’t mind criticism – I never panned people just because they downvoted my app – some app reviews manage to rub me in exactly the wrong way.

It’s not about the rating, but about the attitude some people feel entitled to put out, which are made worse by the “free stuff” context.

So – just to let some steam out and have a laugh- here is a small collection of moronic behaviors I find particularly irking on the store.

The carrot and stick review.

The basic idea that rattles in the reviewer’s skull is that you – the software developer – will do anything short of straight prostitution to get five stars on the store:

⭐☆☆☆☆

Nice app, but there is no “X.” Add “X” to get 5 stars.

Where “X” is a feature, not a bug (otherwise the user would be right, and I usually don’t mind at all!).

Let’s break this down: the user likes the app – he writes that in the review – but since in his delusions you are his bitch, he downvotes it anyway.

“X” can be just anything. From a trivial change to some sort of monumental update that will twist the app into something completely different.

Maybe he downloaded your video player but what he really wanted was to play chess.

The change could also require just a minor edit, or it could take weeks worth of work.

He doesn’t care that the app is free and that you have a daily job and a family: if you put out a month of your life, he will come back and give you 4 extra juicy stars.

But here is the cherry on the cake: even if you add the feature he wanted all along – as sometimes happens by chance – he never does

The “your app broke my cat” review.

These reviews are always written with alarmistic tones and look like the ravings of a lunatic:

⭐☆☆☆☆

DON’T INSTALL THIS APP I INSTALLED THIS APP AND MY PHONE DONT WORK ANYMORE STAY AWAY!!!!!!

I titled this kind of review “the app broke my cat” because this is just as consequential as the reviewer’s argument.

What is the user thinking, I wonder?

Does he think that I’m some kind of demigod that can ply his hardware to my evil will?

Or that my ineptitude is just as powerful and it can have almost reality-bending effects

Either way, I’m kind of flattered by this kind of review, really!

Let’s put aside that damaging a phone with a user-level app is an arduous accomplishment at best, “VR Theater” has more than a million users and a decent rating.

Don’t you think that if it was meant to destroy smartphones, someone else among that million – other than you – would have noticed?

The “I don’t speak English and it’s your problem” review

It’s always (and comprehensibly) written in another language:

⭐☆☆☆☆

Qo’noSDaq tlhInganpu’ [no Klingon translation]

So let me get this straight: I don’t expect anybody to know English (let alone Italian). 

The app has just those localizations because these are the only languages I know and I don’t want to put out money for more translations.

On the other hand, since the user didn’t bother to learn a single word of English, he expects that I should learn the tongue of Mordor (or whatever…), or it’s 1 star.

There is a semi-legitimate reason for these reviews.

I publish the app on all stores worldwide, and (I presume) the app description is automatically translated to the recipient’s language, which may give them the illusion that VR Theater could also be translated.

Still… it’s a video player, not a text adventure. Do you really need a translation for the word “Play”???

Also, I don’t expect a good review but how about “no review at all”?

The “useless” review

They come in various shades of irritating and I treat them accordingly

There is the banal:

⭐☆☆☆☆

useless

which receives a neutral or even apologetic answer: it may be rude, but it’s a borderline case of fair criticism.

Then there is the “people are assholes” version:

⭐☆☆☆☆

crap

which is still tolerable, if not else because I got to learn every possible declination of the word “shit” in every possible language.

And then there are those like this one, that I just received today:

⭐☆☆☆☆

Useless app ! Do nothing. Only make the things very small to see. Can damage eyes.

and checks all the boxes for:

  •  unfair
  • defamatory
  • deranged.

It’s the same old story:

A guy meets a VR app, the VR app requires a VR headset to work, the guy keeps staring into the VR app with his naked eyes, the guy blames the developer.

Note that the app is called VR Theater, and it has a Cardboard VR headset on the store banner and on its very icon.

And, again: thousands of people found the app useful, isn’t the faintest clue passing through your mind that maybe – just maybe – you are not the only genius to have it figure it all out, but you just have no clue?!

The paranoid review

Big Corporations, the Illuminati, reptilians. 

This user knows everyone is after him and that I – with my app – am up to my knees in the great conspiracy.

⭐☆☆☆☆

WHY DOES THE APP NEED TO ACCESS MY FILES? THIS APP STEALS DATA.

I’ll tell you “why”.

It’s because VR Theater is a video player. It plays your videos. Your videos are stored as files on your phone, and my app needs to access them, to play them.

But no, he is not buying it! He has it all figured out, and no one will steal his data, not on his watch!

 Finally, I found a worthy opponent! 

And this is what he does: he logs into his Google account – which is associated with his real name, phone number, and email (the same account that tracks all his web traffic and probably his position) – and he downvotes the poor fucker who writes software in his spare time because clearly “he is after his data”.

He will probably brag about it on Facebook too.

Closing Remarks

This post is about stupid reviews, not about stupid people.

If you want, it’s about “people who occasionally do stupid things and write stupid reviews”.

So, while I hope you had some fun reading, I always try to keep in mind that technology is widespread – which is a good thing – and all sorts of people own a smartphone nowadays.

There are…

  • kids
  • elderly people
  • people from unprivileged classes or communities
  • just some random dude who’s always nice but had a terrible day and lost his temper for just that one moment.

…and none of them deserves hate or contempt for a review. 

Heck, my mother could have written one of those (and God knows if she can be a bitch sometimes).

Then of course there are the jerks that may deserve contempt but, you know… being one that writes posts like this, who am I to cast the first stone?!

Good fences make for good neighbors

If you work in an environment as professional as mine, automatic testing is probably a given.

Odds are, also, that the expression “Design by Contract” (or DBC) isn’t. Maybe that you even never heard about it.

So, what is it?

DBC, in short

If you want to know more about “Design by Contract” and “Assertive Programming” in general,  “The Pragmatic Programmer” (Thomas, Hunt) has a section about each to get you started.

Design by contract is a form of assertive programming where you ensure that functions/methods only operate on sound input and produces the output expected.

You can also identify “class or state invariants”: conditions that should hold before and after each function/method execution, and enforce them.

How do you implement it?

Some languages, like Clojure, have built-in support for DBC (I have briefly mentioned this technique in a previous entry on my blog), for example:

(defn function [arg1 arg2]
    {:pre [(is-foo? arg1) (is-bar? arg2)}]
     :post [(looks-right? %) (sparks-joy? %)]
    ;; your code here
)

will throw an exception if the checks on arg1 or arg2 fail, or if the function result will not "look right" or does not "spark joy".

In other languages, you can still make do. The first time I used DBC was with Objective C on an iPad application. Since the language doesn’t support it, I used a mix of assertions and preprocessor macros.

If I had a function and I wanted to make sure it would add a single value to a non-empty array, I would have done something like:

// PSEUDOCODE!
void appendValue(NSMutableArray *arg) {
#ifdef DBC
    NSUInteger origSize = [arg count];
    assert(origSize > 0); 
#endif

// Do your thing

#ifdef DBC
    assert(origSize + 1 == [arg count]);
#endif
} 

It’s clunky – I know – but it works, and proved invaluable thorough development.

Plain Java can get something similar using the assert construct but is limited by the extensive use of mutability (just as in the previous example) and lack of preprocessing.

Why use it?

First of all, if you are thinking now: “Well, I do TDD or UT, I have full coverage: I don’t need DBC!” you would be factually wrong.

DBC is complementary to unit testing, and it provides value whether you have a testing infrastructure in place or not.

DBC tests the application while it runs, where UT – even a comprehensive end-to-end test – will only check a selected scenario.

One of the main strengths of UT (at least design-wise) is that it’s a form of black-box testing: it ensures that the test subject produces the expected outcome in a set scenario, without delving into the object’s state.

DBC, on the other hand, ensures that your application is internally coherent at runtime as it checks the internal state, and it’s an inherently low-cost approach.

Consequently, you can easily practice DBC in situations where UT and TDD have a prohibitively high cost/benefits ratio, like legacy code or poorly designed applications.

Why is DBC not popular?

If DBC is so good, why isn’t it as popular as TDD or UT?

One of the reasons – I think – is that Test-Driven Development and Unit Testing are both safety nets and design techniques.

DBC – on the other hand – helps the developer to think about the scope of each function but does not lead to clean code.

I suspect – however – that this is only part of the reasons.

I think the main reason why we don’t use it is that – as developers – we tend to be complicit with our code.

It’s the same reason we don’t do assertive programming: we are deeply uncomfortable with the idea of making our code crash, even if we are sure it’s not doing the right thing.

We root for our code, beyond reason.

We think:

Wait…

Is my method unexpectedly receiving an empty list?!

Well, let’s still give it a chance!

Go on anyway, "boolean checkNonEmptyList(List arguments)": I believe in you!

Exemplifying the first week of my current job, I was doing pair-programming with a senior team member. The guy is reviewing my code.

At some point, he sees a (rather tame) assertion I had put in one of my methods. He looks at me aghast and mutters:

We don’t put assertions in the code. We don’t do this.

As of now and that I know, assertions are still taboo in my workplace (a workplace which is – to be crystal clear – a place full of very respectable and capable developers).

Checks are still made of code

I’ll play the devil’s advocate at this point.

There is at least a good reason to look at DBC with distrust: the checks you are adding in your code are – of course – still code.

And like any code, it will contain bugs and it will fail on occasion, so the idea of adding a check that may actually add a bug to your code does not win a popularity contest.

Sure, you can be careful all you want but, eventually, your application will crash due to a bogus contract.

Is then DBC still worth it? YES. The number of bugs you’ll catch with DBC fairly exceeds the bugs you will introduce, and finding one of the latter contributes to your understanding of your code.

Yet, this is probably a (more reasonable) explanation of why this technique is less used and why I respectfully disagree with those who would leave all DBC checks on in production by default.

Conclusions

“Design by Contract” should be in every programmer’s toolbox: it forces you to think about the scope of your code, provides runtime testing, and is cheap to implement.

Like everything in this world, it isn’t devoid of downsides, but the benefits far outweigh the risks.

My “Advent of Code 2020” with Clojure

This is the last of a three parts series of posts about the “Advent of Code”, you can check this post if you want to know about the initiative or this other one for my general experience with the 2020 edition.

TL;DR: using Clojure, I loved the flexibility of the language: solving the problems with a functional paradigm added another (welcome) layer to the challenges. The lack of ready tools has been a minor disappointment, though.

As usual, I hardly even consider non-LISP languages for any leisurely coding, so I picked Clojure for my first participation in the “Advent of Code”.

My choice assumed (correctly) that the competition was comprehensive enough to allow developers with languages as slow as Java to have a chance: anything else would have made “Advent of Code” even more niche than it already is.

The next paragraphs summarize my experience with the “Advent of Code” using Clojure. Enjoy!

Join the dynamically typed side!

You chose Clojure because it’s not Java, then why coding in it as if it was Java?

It’s been a while since the last time I coded in Clojure, and, being a Java developer by day, I felt some (unexpected and unwelcome) uneasiness passing generic vectors and maps around.

At some point, I gave up on the temptation, and I did reify a protocol.

Around day 20, after creating the umpteenth protocol (because what sense does it make to create the C protocol if there are not the A protocol, the B protocol, and since we are at that, the D and E protocols?), I trashed the whole thing.

I restarted from scratches, this time using generic data structures, and never looked back.

Darth Vader "Join Me"

Now, I’m not saying that protocols (and classes, etc…) don’t have their place, but a competition based on daily, ever-changing problems, ain’t it.

And with Clojure, you have a choice.

Functional thinking

aka “that thing I don’t (yet) have”

I guess that if you really miss your C programming and want to make a trip on memory lane, you can start using atoms and do blocks like crazy and, besides producing ugly code, no harm will be done.

If you aim to use Clojure idiomatically, though, you’ll have to solve familiar problems in counter-intuitive ways.

This is a challenge I picked up willingly.

It certainly made me slower but provided some of my best return-of-investment from the competition in technical terms.

It also set my mind at ease about some worries I had about performances: transposing a table using map and apply operations is not as slow as I imagined it to be.

Planning for failure

Mh… It’s not working… Should I have written some tests?

I don’t think so: the scope of each problem was so small that the overhead of testing or TDD didn’t seem to pay for itself.

Also, I can be boring up to a point.

On the other end, not having a safety net (especially with a dynamically typed language) can be a source of minor headaches.

One solution that complements unit testing, is Design By Contract

Have you heard about "Design by Contract"?

I am by no means knowledgeable on the matter, I picked what I know from the excellent book “The Pragmatic Programmer” (Thomas, Hunt – new or old edition, it doesn’t matter).

It boils down to having runtime checks – mostly around functions – to ensure each method receives just what you expect and produces the right outcome.

You can do that in any language but – it turns out – Clojure conveniently supports it with the :pre and :post conditions in the defn special form.

In this (rather uncalled for) use of them:

(defn mangle-list [numbers]
    {:pre [(every? number? numbers) 
           (not (empty? numbers))]
     :post [(number? %)]}
    (reduce * numbers))

the :pre condition makes sure that the input is a non-empty sequence of numbers, and the :post ensuers that the output is a number.

It won’t make a difference with simple examples like this one, but when you are knee-deep in code, pre and post conditions can save you a lot of time.

Tools are not at hand

Let’s profile this thing with VisualVM! (5 minutes later…) Nevermind.

Debugging and profiling with Clojure is not as ready and convenient as with Java. This is a fact.

I’m not saying it can’t be done, I’m saying – again – that it’s not convenient, not at hand (cf. “Simple made Easy” by the man himself), that’s all.

Lack of tools is the most relevant grief I have with the language, even though the language itself is (mostly) not to blame.

That said, if you want to set a breakpoint, CIDER works beautifully (and I absolutely love CIDER, mind you!), but if you want to break on an exception you are out of luck.

On the other hand, you can use VisualVM for profiling, but at the first glance, if you don’t exclude the Clojure namespace first – between that and REPL-related threads – you have to wade through a lot of garbage to find what you are looking for.

Spiderman: "Nevermind, I'll just go..." meme

Conclusions

I had other possible choices for the competition.

Gambit or Guile would have been some excellent candidates if execution speed was of the essence, but I am frankly not proficient enough with them, and they don’t have the JRE with its goodies backing them up.

Emacs Lisp holds a place in my heart, despite being ancient, but the awkward support for dictionaries (or as they call them: “tables”) would have made everything so much harder.

The possibility of picking an entirely new language (Elixir is next on my TODO-list) crossed my mind and was – fortunately – crossed itself. I really dodged a bullet there (not because of Elixir specifically!).

I chose Clojure, fortunately, and – despite some minor hiccups – the combination of REPL-driven programming, support for a rich set of collections, and dynamical approach turned out to be a winning move.

My “Advent of Code 2020”, in review

If you don’t know what the “Advent of Code” is, you can check my previous post on the topic.

Getting in the top 100 requires a kind preparation, and mindset I don’t have.

It’s the kind of skill you train when you routinely compete in online events where the super-fast win and, while I respect it, it’s not the approach I like when coding.

I golf to be deliberate and thoughtful, and coding clean instead, so I took the competition at my own pace, with only one exception: I tried to solve each problem in its own day.

Here is my take-away on the competition itself, which I’ll hopefully follow with an entry about doing “Advent of Code” with Clojure.

Even one problem a day can be too much

Having a regular job and a family, I often started the day’s problem at 10PM, only to finish it at 3AM.

Maybe other people are faster than I am, but for as long as I managed to ride the wave of each daily problem, it’s been taxing.

It’s fun (most of the time)

You wake up and you are being treated with a new, absurd MacGyver-esque problem. You wrap your head around it, find an elegant solution, and give yourself a pat on the back.

Some problems required more time to crack than others, it’s hard work sometimes, but the lasting feeling is rewarding.

This is what happens on a sunny day.

It’s less fun when you remain stuck

Day 23 turned out ugly. I got seriously stuck for two days on a problem that – I realized afterward – was embarrassingly simple.

From the statistics on the website, I know all other participants are moving forward. The leaderboard doesn’t lie: I’m among the few who are finding that problem a challenge.

At some point, I literally felt like the last sucker: I personally questioned my value as a developer. And I was still stuck, so I had to bite the bullet and check Reddit for other people’s solutions.

It turned out the problem was among the simplest ones, and right because the solution was so simple, I had pre-emptively discarded it a priori.

I won’t lie: It did hurt. A LOT.

Finishing the remaining two problems 2 days late didn’t really deliver much joy or a sense of accomplishment, but I tried to put the experience in perspective.

In hindsight, there are some considerations that help to mend my ego a bit, but that can have some sort of general value.

First, I may have been trailing after the group, but it was still a small group of people who don’t just code but also love so much doing so, it solves coding problems while on vacation. So yes, I may even have been “the last”, but I’m definitely not the last sucker.

Second, of the almost 150000 people who started participating only 8% managed to finish it: so even if I was among the slowest 10% of people to solve day 23, I’m in the top 8% who didn’t give up!

The best day was the worst day

More importantly, however, day 23 has been the most enlightening day.

Maybe the other ones may have been more pleasant, and they may have even managed to let me think and improve a bit, but thedebacle of day 23 highlighted a fundamental flaw in my way to approach problems in general, and hopefully managed to get the point across.

It did also show my shortcomings in responding to failures, in general: letting myself feeling down, grudgingly pressing on, losing the sense of perspective are all pains I could have made without.

I picked up the (wonderful) “Thanks for the Feedback: The Science and Art of Receiving Feedback Well” (Douglas Stone, Sheila Heen) again to re-read the chapter about “giving yourself a second score”.

The idea is that failures are not always under our control, but the way we react to them is and, like in this case, so it’s the way I react to my failure to react to failure.

I also refreshed a bit another favorite book of mine: “The Subtle Art of Not Giving a F*ck, A Counterintuitive Approach to Living a Good Life” (Mark Manson).

I didn’t re-read it all, I just picked two sentences:

“Pain is part of the process”

and

“Failure is the way forward”

And only God knows how much pain and failure I face every time I code…

About the “Advent of Code”

Ok, it’s too late now, but say it’s Christmas: vacations are looming, plenty of time to focus on your project, but what if you don’t have one?

Eric Wastl must have felt this pain: he’s the mind behind the Advent of Code competition, a friendly and geeky website where each day of the advent a new computing problem is unlocked.

The puzzles are bound together with a backstory: maybe you are rescuing Santa and saving Christmas or – like this year – trying to make it to your resort for a vacation.

It doesn’t matter: whatever the backstory could be, you will end up facing new zany situations every day, the solution of which will require (you guessed!) computing and logical skills.

The puzzles have two parts: the first one is usually straightforward, while the second part can be tricky: it may require some specific optimization techniques, some math, or just intuition.

Each problem/day solved, a new part of the ASCII-art calendar is also unveiled: a thing of irresistible geekish and childish beauty!

The narrative is witty and fun, and the problems are engaging (most of the time). Old calendars remain open, so you can still solve all the old challenges.

Even if it’s not the advent anymore, you can still wake up, read a problem, and set yourself the goal of finishing it within the day, maybe competing among friends on a private board (the site has those).

It’s a great project: I’m impressed about how much care has been put into the narrative (even if it’s absurd…) and the details, so whether you are bored or are looking for some new kata to practice a new language, make sure to check it out!

The only winning move is not to play

I have a mixed relationship with Video-games: on one hand, they are a welcome stress vent and have inspired me to learn and become a programmer; on the other one, they are a black hole sinking huge amounts of my spare time and energies.

JYDGE by developer 10tons is my latest addiction: it could be superficially categorized as a twin-stick shooter, but it’s a rather new blend of action and stealth with puzzle elements.

One thing buggers me, though: in order to reach a specific secret level, you have to beat a challenge that requires you to score more than 60k points in a tiles-matching gem game called “Gembine”, and it’s rather hard.

You can see a short game here:

it’s from another 10ton’s game – Crimsonland, they embedded it in there too – but the differences are only aesthetic.

The hallmark of the best projects

I love 10tons’ games, yet I don’t share their love for gem-matching.

If I wanted to play gem matching games, I would have probably downloaded a puzzle game in the first place, which is why spending my time mastering Gembine is not an option.

What I would love to do, is to create a program to solve the game for me!

Due to its simplicity, Gembine is an ideal problem for learning a number of topics but, more than anything, it has the most desirable trait of all for a learning project: it’s a fun, silly thing, and when learning is involved nothing beats a task that’s nothing about time pressure and destination, and all about the journey and the fun of doing it.

If you have to learn something, fun is the best way, and if you are not enjoying yourself, either be sure to have a very strong motivation or don’t even bother trying.

What I want

I want a program that interfaces with Gembine and beats it in a reasonable time. Even some hours or a night of automated play are OK.

I watch a movie or go to sleep, and the next day the secret level is there! I need to top 60k points for that.

Just for comparison, my highest score has been about 53k points: I made them on the first play-through, at 2 am – completely groggy – and I never managed to get even near to that ever again.

My average game is between 15k and 25k. A long way to go to 60k.

The game rules

The game rules are simple: you start with a 4×4 board with some gems, and your objective is to move the gems on the board, as a whole, in a specific direction, to overlap gems of the same type. You can check the gameplay video embedded above to get the idea.

When two gems of the same type overlap the gem you moved disappears, the other one gets promoted to the next level, and you score some points.

There is a catch, however: each time you make a move, the computer will add a random gem on the side of the board you move from, which means the player should match on average a gem per move to keep up.

As soon as there are no available moves, the game ends.

Tools of the trade

As a language, I pick Clojure; it follows my gradient of maximum fun, but it’s also a natural fit: it’s JVM based, and has a REPL.

There is a nice class in AWT called Robot that allows you to emulate key presses: let’s take it for a spin!

$ lein repl
[text blurb]
$ (import [java.awt.event KeyEvent]) ⏎
$ (doto (java.awt.Robot.)
(.keyPress KeyEvent/VK_UP)
(.keyRelease KeyEvent/VK_UP)) ⏎
$ (.keyRelease KeyEvent/VK_UP))

It works! The REPL has a history, so when the first “doto” is executed, the program taps the up key, and I get rewarded with the last executed line (the keyRelease one).

A second test, with an initial delay, allows me to switch application, and ensure that the key event is really sent to whichever window has the current focus.

Not bad for 5′ of work, isn’t it?

Monkey Business

Gembine is not picky about trying an illegal move, so I have enough ammo to create a blind random player.

I create a new Leiningen project for that, cram together a function that taps a random arrow key with some delay and, voilà! The blind random player scores 11K after about 150 moves.

The score is almost in the range of my average game, which means I’m about as good at Gembine as a random moves generator.

This hurts.

Let’s put pride aside for a second and analyze the result: what I have is the algorithmic equivalent of putting a monkey in front of a typewriter to write a Shakespeare’s drama, and just as inefficient (if less smelly).

Monkeys on typewriters comic

Theoretically, this could solve the game  in a finite – albeit very very long – time.

What I really need is a smart program that can play strategically, and for a smart program to be useful, it has to see what’s going on, to see the board.

I don’t need to google for the tool to use for this: OpenCV for the win!

OpenCV

I will not bother you with the fine-grained details of setting up OpenCV on my system.

Let’s pretend it’s a cooking TV show: the host mixes flour, milk, baking powder, and sugar, puts the content in an oven, and a second later, voilà: a beautiful cake!

Too bad that the cake was cooked 4 times, that some people got poisoned and 2 of them died: what happens in the backstage remains in the backstage.

Just a couple of random remarks, since compiling and installing OpenCV properly and turning into a couple of JARs has been a huge pain in the ass:

  1. don’t be smart: follow the OpenCV instructions for java compilation closely, there is even an official Clojure/OpenCV tutorial. Do read it.
  2. if you are on Linux, make sure to run execstack -c on the java .so file if java complains even slightly.
  3. don’t mind what you read around: just because you are able to
    (clojure.lang.RT/loadLibrary org.opencv.core.Core/NATIVE_LIBRARY_NAME))

    doesn’t mean shit.

  4. automatize as much as you can because, unless you are a luckier bastard than I am, you are going to work on this stuff a lot.

Long story short: I have two custom made JARs in my local MAVEN repository – one for the java bindings and one for the native library – and by adding to my project.clj

:dependencies [[org.clojure/clojure "1.8.0"]
               [commons-io/commons-io "2.6"]
               [opencv/opencv "3.4.1"]
               [opencv/opencv-native "3.4.1"]]

and after loading the native OpenCV library in my REPL

> (clojure.lang.RT/loadLibrary org.opencv.core.Core/NATIVE_LIBRARY_NAME))

I can – for instance – load an image with something like:

> (org.opencv.imgcodecs.Imgcodecs/imread "resources/mocks/gembine-1.png" 
                                         Imgcodecs/IMREAD_COLOR)

This last test is important: you’ll read around that when you are able to load the native library you are all set, but that’s BS.

In some occasions, my installation was good enough to load the libraries and create a simple instance (like Point) but failed to load an image.

Finding images in other images: template matching

I need to understand which gem – if any – is present at each board square.

This is a representative Gembine’s screenshot:

Screenshot from Gembine, taken from 10tons’ JYDGE

There are two different gem colors (red and green) and five different shapes (small sphere, normal sphere, triangle, square, and pentagon).

They can appear on each square of a fixed 4×4 board, or where the “Game Over” text is.

What I need is template matching: searching an image into another.

If you want to delve deeper you can start from OpenCV’s official documentation and possibly continue with this chapter on convolution from Steven W. Smith’s excellent “Engineer’s Guide to Digital Signal Processing”.

I need to experiment a bit, and while often REPL programming is enough, it’s a good time to put a testing infrastructure in place.

Clojure has an excellent support for TDD and UT, I decide for a data representation for the board, like:

[[:rB :rt :rB :rb]
 [:gb :rB :rs :rB]
 [:rb :gB :rp :gt]
 [:rB :rb :rs :rb]]

(it represents the screenshot above), then with:

(ns clojure-gembine.board-test
    (:require [clojure.test :refer :all]
              [clojure-gembine.board :refer :all]
              [clojure-gembine.test-utils :as utils]))

(deftest test-read-board
    (testing "result on mock 1"
        (is (= [[:rt :rB :rt :rs]
                [:rB :rt :gb :rB]
                [:rb :gB :rb :rs]
                [:rs :rt :gB :rb]]
               (read-board (utils/load-mock-image 1)))))
    (testing "result on mock 2"
        …)
…)

I can tweak the search algorithm to my heart’s content.

When I finish, my program has a pair of eyes but still needs a brain.

Minimax

First I have to simulate the behavior of the board.

I have left my iPad Pro with its pencil in the Ferrari, so I have to make do with a piece of napkin.

iTableCloth, now with pressure-sensitive writing and tomato sauce stains

I can use Gembine’s symmetry to reduce the game to a single move – moving the tiles from right to left – while using rotations to simulate the remaining three.

Armed with this knowledge and a solid TDD approach, it’s just a matter of shedding a tear for the poor sods who tackle this with plain Java, and I’m ready for the actual algorithm.

I settle for a simple Minmax implementation – or more precisely a Negamax – that looks ahead one move, and see how it fares.

Minimax needs a score function.

I want one that embeds some knowledge about the game and its strategy without biasing Minmax into playing a self-defeating game: in this way, the single depth minimax should perform a little better than could be reasonably expected.

At least, this is the official reason.

A less noble reason is that I don’t have an exact representation of the score (and I have no intention to put elbow oil in it), so this crackpot theory serves me well.

My score has two components: the first correlates to the score proper, and it’s a weighted sum of the empty spaces after a move; the second count the number of adjacent gems of the same type.

First Results

Minimax level 1 implementation beats my 54k score at the second or third attempt and reaches the secret area with 82k points in the first handful of games.

BINGO!

You can still follow the moves logic, roughly speaking, and I confess that – while I’m watching the game – I feel proud.

Yet I’m not satisfied: I want to know what happens after two green pentagon gems are combined!

Pen, paper and a bit of logic suggest that there should be another color, and I want to see it.

Thanos and his glove can eat my shorts

Level 2 minimax seems to work better than level 1 (it should, in absolute terms), but it’s hard to tell.

I create a Gembine engine in Clojure and use it to score my solvers.

My ersatz Gembine (I looked it up on Wikipedia weeks ago and I couldn’t wait to use it: it means “inferior copy or replacement”) doesn’t have a concept of “score”, but I can do statistics counting the moves before game-over for each algorithm.

Minimax level 2 clearly outperforms level 1, and my spiked score function seems statistically better than a more conventional one.

My games end too often in the secret level now, so I teach my code to detect it and restart Gembine, and I make it run unattended for a whole night.

And this is the result:

gembine screenshot with blue gem
A rare screenshot of a blue gem in its natural environment

Putting the awesome score aside, with an unforeseen stroke of luck I have been able to take a screenshot of a blue gem!

In one of the games, my program got stuck just before a game over: it couldn’t recognize the blue gem correctly – I never taught it to do that – so it tried to merge it with a green one

The game was almost lost, but I managed to scavenge a score of 120.100. Quite impressive, if I’m allowed to say it myself.

As a last experiment, I generalize the minimax algorithm to take the depth search as a parameter.

Going as deep as four moves proved to be impractical for performance reasons: it took a whole night to play a game and a half and, while I could probably optimize the code a lot to take advantage of all my CPU cores, it would still be slow.

Regardless, Minimax level 3 alone delivered me some satisfaction:

177.600 High Score image
Cha-Ching!

You know what? I think I can settle with that.

The code

You can find the source on GitHub

It has some instructions of sorts but it’s a prototype, so keep your expectations very very low.

Running the program from the command line you will be bound to the level 1 minimax, and only the truly worthy will be able to master the code’s secrets and tap from all minimax depths…

…or the ones that will copy this in a Leiningen REPL:

> (utils/sleep 10000)
  (execute-moves move-with-logic 
                (move-logic/create-deep-minimax-solver 1024) 
                 true)

That’s a minimax of depth 1024 for you, baby!

See you in 2300AD, and you’ll tell me how your game went.

Say you want to draw stuff…

I recently had to deal with troubles involving Swing and the Event Dispatch Thread (EDT for friends) at work.

Details are shrouded in secrecy, and flying monkeys will be rightfully dispatched, should I delve on them, but it suffices to say, they had to do with graphics and EDT concurrency.

Plain Java solved all those problems alright, mostly with the judicious use of lazy initialization and and the invokeLater utility method and yet, despite the whole bunch of Java 8 features, the implementation feels a bit clunky, wherever anything, but the shortest method references, is to be used.

Incidentally, I started working on graphics at home, and ended up with similar issues, but this time, with Clojure.

While Clojure can’t do anything unique or special, when compared to Java, it provides tools that allow to actually play with graphics, and solve some of the thread-related issues more elegantly.

Let’s play a bit with Graphics on Clojure, shall we?!

REPL-oriented graphics

Clojure coding is not Clojure coding without a REPL running, so let’s start one (maybe in CIDER mode?), and see if we can create a small frame to write into…

> (def test-frame (doto (JFrame.)
                          (.setSize 320 200)
                          (.show)))
#'some-namespace/test-frame

And a brand new Frame appears, ready to do our bidding:

easy as pie!

Let’s scribble on it: I’m not Pablo Picasso, so I’ll just change the background (you’ll see a lot of background changes, as a remainder that – as I have just mentioned – “I’m not Pablo Picasso”).

  
> (.setBackground (.getGraphics test-frame) Color/RED)

And surprise!

 

…nothing happens.

Each JComponent has its own paint method, which is called at every repaint and will happily overwrite whatever we throw at the object with our flimsy drawing attempt.

Fat bastard with 'you bastard' caption

 

We could override the base paint method, and create a new object every time we want to experiment, but honestly this is sooo 1995!

Even vanilla Java coders have better means now, with the judicious use of a Consumer<Graphics> interface, so we’ll do pretty much what they do nowadays:

> (defn show-frame
  ([width height] (show-frame width height nil))
  ([width height paint-f]
   (doto (proxy [JFrame] []
           [paint [graphics]
            (if paint-f
              (paint-f this graphics))])
     (.setSize (Dimension. width height))
     (.show))))
#'some-namespace/show-frame 

show-frame accepts an optional method that will dictate how the component is painted.

Passing a method reference will ensure that changing the paint method definition will be “seen” by the class:

>  (defn my-paint [this graphics] (.setBackground this Color/BLACK))
#'some-namespace/my-paint
> (def frame (show-frame 320 200 my-paint))
#'some-namespace/frame

Good enough:

black JFrame

Let’s see what we can do interactively:

> (defn my-paint [this graphics] (.setBackground this Color/MAGENTA))  
#'some-namespace/my-pain
> (.repaint frame)
nil

 

Et voilà: interactive REPL scribbling enabled (the annoying color is a bonus)!

magenta JFrame

And yet I might have forgot something…

Every Damn Time

It’s counter-intuitive, but even JComponent instantiating – no matter how basic the component is – should go into the Event Dispatch Thread, as this Stack Overflow answer points out.

Luckily enough, we Clojure programmers can borrow SwingUtils.invokeLater and SwingUtils.invokeAndWait, so no big deal, right?!

> (SwingUtilities/invokeAndWait (fn [] (show-frame 320 200)))
nil

The frame appears, but the syntax is a bit on the hideous side, so let’s sweeten it a bit:

> (defmacro invoke-and-wait [ & forms]
       `(SwingUtilities/invokeAndWait (fn [] ~@forms)))
#'some-namespace/invoke-and-wait

Which enables us to a slightly shorter form (which can be incidentally used with multiple forms or without one too):

> (invoke-and-wait (show-frame 320 200))
nil

Let’s see: we are able to created our frame in the right thread, and to draw on it interactively, so home base!

WAIT A SECOND!!!

The invokeAndWait method returned nil!

Despite being the SwingUtils.invokeAndWait a synchronous method, it doesn’t accept a supplier and return a generic as it could, but — probably for historical reasons — it operates on a Runnable and returns void.

This is OK for the typical use, that is, to trigger some UI change that has only side effects, but what if we want to use it to create a JComponent interactively?

Here is the answer:

>  (defmacro invoke-and-wait
    "Invokes the forms within an invokeAndWait method, and returns the value of the last form."
    [& forms]
    `(let [result# (atom nil)]
       (SwingUtilities/invokeAndWait (fn [] (reset! result# ((fn [] ~@forms)))))
       @result#))

With this new macro, (invoke-and-wait (show-frame 320 200)) returns the frame, which we can store wherever we want, for later use.

The promise of some very Graphics content

What if we don’t want to use the EDT thread in lockstep?

Abusing invokeAndWait leads to unresponsive interfaces: we want to take advantage of both the thread safety AND the concurrency EDT would enable us to.

We want to eat the cake, we want to have it, and we want it now.

We can use a slightly modified version of the last macro to invoke the invokeLater returning a special value (that is, a promise).

The promise will block until a value has been delivered to it, when de-referenced:

  (defmacro invoke-later
  "Invokes the forms into a invokeLater, returns a promise with the result of the last form"
  [& forms]
  `(let [result# (promise)]
  (SwingUtilities/invokeLater (fn [] (deliver result# ((fn [] ~@forms)))))
  result#))

Invoking the creation of a frame will still do the job, and return something like:

> (invoke-later (show-frame 320 200))
#promise[{:status :pending, :val nil} 0x248259a6]

Which will block any attempt to use the value, until the operation is completed, and the value properly realized.

This last macro is also rather general, and it will make the invoke-and-wait one actually redundant, as the latter can be reduced to a call of this one, followed by a @ operator on the result:

>  @(invoke-later (show-frame 320 200))
#object[clojure_scratchpad.graphics.proxy$javax.swing.JFrame$ff19274a 0x7a35ae23 "clojure_scratchpad.graphics.proxy$javax.swing.JFrame$ff19274a[frame13,0,27,320x200,layout=java.awt.BorderLayout,title=,resizable,normal,defaultCloseOperation=HIDE_ON_CLOSE,rootPane=javax.swing.JRootPane[,5,25,310x170,layout=javax.swing.JRootPane$RootLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]"]

Now we can play interactively (and safely) with the Graphics2D class, and experiment with the code — as it suits to all good Clojure programmers — as I did with this example:

Green still life painting

 

Nope. This is an actual Picasso (“Green still life [1914]”)…

So now what?

“Profanity is the one language all programmers know” is something I picked in a book months ago, and it’s been in my head ever since.

I think it summarizes at least 90% of software development, when you are lucky enough to take on new challenges.

The 10% is the carrot on the stick, when – if – everything finally works, and you feel proportionally rewarded for your efforts.

This blog is mostly about that, about the struggle to get things done, and about the profanity (but with the profanity omitted).

A mixed bag of accounts about books I have read, alarmingly esoteric technical challenges, and the occasional – and shamelessly biased – remark on new trends.