Miro's NotesZola2024-01-08T00:00:00+00:00https://mirosval.sk/atom.xmlDepending on Apple Frameworks in your Nix Flakes2024-01-08T00:00:00+00:002024-01-08T00:00:00+00:00https://mirosval.sk/blog/2024/nix-apple-system-frameworks/<p>You may find yourself faced with the following error: </p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>❯ cargo build
</span><span> Compiling project-name v0.1.0 (/Users/mirosval/Projekty/test)
</span><span>error: linking with `cc` failed: exit status: 1
</span><span> |
</span><span> = note: LC_ALL="C" <omitted>
</span><span> = note: ld: framework not found Security
</span><span> clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
</span><span>
</span><span>
</span><span>error: could not compile `project-name` (bin "project-name") due to previous error
</span></code></pre>
<p>It may be concerning some other Apple Framework, most commonly I've found it to be either <code>Security</code>, <code>Security</code>, <code>SystemConfiguration</code>, <code>Accelerate</code> or else it's one of the other <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/darwin/apple-sdk/frameworks.nix">Apple Frameworks</a>.</p>
<p>This happens because your Nix shell is isolated from the rest of the system and these frameworks are an undeclard dependency of your project. </p>
<p>So in order to fix it, you need to add the offending frameworks to your Nix configuration (Flake or <code>shell.nix</code>):</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span> </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>;
</span><span> {
</span><span> </span><span style="color:#d08770;">devShells</span><span>.</span><span style="color:#d08770;">default </span><span>= </span><span style="color:#bf616a;">mkShell </span><span>{
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= [
</span><span> </span><span style="color:#bf616a;">rustToolchain
</span><span> </span><span style="color:#bf616a;">darwin</span><span>.</span><span style="color:#bf616a;">apple_sdk</span><span>.</span><span style="color:#bf616a;">frameworks</span><span>.</span><span style="color:#bf616a;">SystemConfiguration </span><span style="color:#65737e;"># <-- HERE
</span><span> ];
</span><span> </span><span style="color:#d08770;">packages </span><span>= [
</span><span> </span><span style="color:#bf616a;">cargo-watch
</span><span> ];
</span><span> };
</span><span> }
</span></code></pre>
Controlling macOS Settings with nix-darwin2024-01-03T00:00:00+00:002024-01-03T00:00:00+00:00https://mirosval.sk/blog/2024/macos-settings-with-nix-darwin/<p>There are <a href="https://mirosval.sk/blog/2023/nix-getting-started/">many ways you can use Nix</a>. But if you're on a Mac and you want to manage the system with Nix, you need to use <a href="https://daiderd.com/nix-darwin/">Nix-darwin</a>.</p>
<p>One of the biggest advantages of using nix-darwin is the ability to actually configure your system settings with it. However, this presents a problem, how do I translate the macOS settings into a nix config?</p>
<p>One way is to browse through the <a href="https://daiderd.com/nix-darwin/manual/index.html#opt-system.defaults.NSGlobalDomain.AppleEnableMouseSwipeNavigateWithScrolls">nix-darwin options documentation</a> and you'll find entries like <code>system.defaults.NSGlobalDomain.AppleShowAllFiles</code> which, if set to <code>true</code> will show hidden files in Finder.</p>
<p>Another way is to use the <code>defaults</code> utility. For example <code>defaults domains</code> will list all the settings domains that exist on the system. These will include all the installed applications and of course the system. You can then use <code>defaults find <keyword></code> to find all the settings with that keyword. You can dump that to a file, change the settings in System Preferences to your desired values and then run the same <code>defaults find</code> command again and diff the two files to find the settings that have changed.</p>
<p>Last way I found is to use <a href="https://macos-defaults.com/">macos-defaults.com</a> which is a nice resource that can help you find the right settings.</p>
Snowflake vs Redshift2023-12-14T00:00:00+00:002023-12-14T00:00:00+00:00https://mirosval.sk/blog/2023/snowflake/<h1 id="snowflake-vs-redshift">Snowflake vs Redshift</h1>
<p>I recently played a major part in a migration from Snowflake to Redshift. I'm writing this for me if I need to reference it in the future, but I hope others can find this useful as well, as there was a lack of similar posts when I was at the discovery phase of the project.</p>
<h2 id="briefly-about-snowflake">Briefly about Snowflake</h2>
<p>Snowflake is a pretty well known piece of software in the data industry. However if you've not encountered it before, it can be a little intimidating. I'll summarize my knowledge on Snowflake here in terms an Engineer should find accessible.</p>
<p>If you're only going to learn one thing about Snowflake, make that <a href="https://docs.snowflake.com/en/user-guide/tables-clustering-micropartitions">Micro partitions</a>. If you have a basic idea about how databases work, this single piece of knowledge takes you 80% of the way to understanding Snowflake. Snowflake lays out data in small immutable chunks, called Micro partitions. As of this writing their size is set to 16MB, which also sets the maximum limit of <code>text</code>/<code>json</code> columns. Ingestion is limited by the speed at which you can create these micro partitions and write them (I believe they just store them on S3 or equivalent). Updates/deletes can be very expensive, because whole micro partitions have to be rewritten to materialize the change. Reads can be quick depending on the ability of the query planner to eliminate micro partitions from scanning. Multiple warehouses can access the same data at the same time. Time travel, zero copy cloning and all the related features all naturally follow from immutable micro partitions, any point in time snapshot is just a set of pointers to micro partitions. </p>
<p>Secondly, it's important to know that databases (storage) in Snowflake are persistent, but data warehouses (compute) are ephemeral. They spin up and down in a matter of seconds and are billed per second after 1 minute. The cost of storage is very low (usually <10% of the total bill). The cost of compute is something you have firmly in your hand by planning scaling policies on your warehouses. Also attributing costs is easy, just assign a warehouse per cost center.</p>
<h2 id="briefly-about-redshift">Briefly about Redshift</h2>
<p>While Snowflake feels right of the bat as a solid database designed by people who knew what they were doing, Redshift feels very much <em>not</em>. To my understanding, Redshift was hastily put together by taking the PostgreSQL query parser and wire protocol (at version 8) and slapping it on a custom query engine. While AWS is making some great work in the recent years, they still have a long way to go. Everything feels ad-hoc and bolted on somehow.</p>
<p>There are 3 main variants of Redshift: Serverless, RA and DC. DC was initially the only option, it stores data directly on the nodes, so if your data grows, you need to add more nodes and then they just sit there unused, so it can be very expensive if you have a lot of data. RA was the second option, this is somewhat similar to Snowflake in that it persists the data on S3 and uses some local caches to speed up, however it still has limitations, e.g. scaling the cluster will take it down for ~15 min. Serverless wants to be even more reactive and decoupled, but still ends up being pretty inflexible and expensive.</p>
<p>Some problems we had with Redshift:</p>
<ul>
<li>Because of the 15 min scaling downtime, we kept it at a fix size, meaning we didn't have enough scale for the ETL at night, but also didn't use the capacity during the day, so we overpaid</li>
<li>Because all the queries go through the Leader, it is the bottleneck, so there are limits to how much you can scale. Some operations can only be performed on the Leader as well, adding to the problem</li>
<li>dbt uses a lot of metadata queries, which are all answered by the leader</li>
<li>Limted concurrency support often lead to circular locks on tables and failed ETLs</li>
<li>JSON support is pretty weak</li>
</ul>
<h2 id="a-word-about-the-scale">A word about the scale</h2>
<p>Without going too much into the details, I'm talking about tens of TB of data and thousands of tables.</p>
<h2 id="the-migration">The migration</h2>
<p>I really like the idea (I think it comes from Martin Folwer's <a href="https://martinfowler.com/books/refactoring.html">Refactoring</a> book) that (paraphrased):</p>
<blockquote>
<p>Refactoring is a 3-step process: (1) Pre-refactoring, (2) Refactoring and (3) Post-refactoring</p>
</blockquote>
<p>What I take away from that is to insert a step into the process that does not change the behaviour at all, but makes the subsequent refactoring <em>easy</em>. In a migration such as this one that is going to touch hundreds of jobs, you really don't want to both change the job code and also the interface at the same time. </p>
<p>You may want to introduce an interface that all the jobs use (pre-refactoring) and then swap that interface from Redshift to Snowflake in a single atomic operation. So that if you mess something up, you can easily revert it. You do not want to maintain two versions of code that differ in more than one simple way.</p>
<p>In our case we have massaged our architecture from ETL to ELT, with a strong emphasis on the fact that the raw data <em>is</em> the ground truth. Meaning that <em>all</em> the later layers of the data warehouse <em>must</em> be able to be completely recreated from the raw data by a series of transformations encoded in SQL. </p>
<p>Additionally, we have ensured that all our raw data is stored in some sensible format on S3.</p>
<p>That meant we could split our migration into 2 phases:</p>
<ol>
<li>Migrate the <code>raw</code> layer</li>
<li>Migrate the transformations</li>
</ol>
<p>Both of these could be executed in parallel, so we had lists of all the jobs that needed to be mirgated and we assigned the migration tasks (which were detailed in advance) to the team members.</p>
<p>The migration of the <code>raw</code> layer was the more difficult bit, because there was some heterogeneity among all the various extractor jobs, but we ended in a better place overall with the codebase thanks to this migration anyways.</p>
<p>The transformation phase was a lot easier, thanks to <a href="https://www.getdbt.com/">dbt</a>. There are packages and even dbt has <a href="https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros">built in</a> some macros that you just need to make sure you're using in your models and they will be portable between various supported warehouses. It was not perfect and I ended up writing some macros, most notably replacing the <a href="https://docs.aws.amazon.com/redshift/latest/dg/JSON_EXTRACT_PATH_TEXT.html"><code>JSON_EXTRACT_PATH_TEXT</code></a> function with <a href="https://docs.snowflake.com/en/sql-reference/functions/get_path"><code>GET_PATH</code></a>.</p>
<p>Finally as a post-refactoring we have cleaned up all the Redshift-specific code that was awkwardly left lying around. In many cases Snowflake has better options for things that Redshift can't do well.</p>
<h2 id="thougts-on-redshift">Thougts on Redshift</h2>
<p>Overall I have used Redshift 2x in my career and both times it ended up not being the right tool for the job. Partly because we chose poorly, but also because I feel the way AWS presents their products can be misleading. </p>
<p>For instance the first time we used Redshift, it was on a path that was hit from UI. Nothing in the documentation suggested what kind of problems we might encounter if we did that, though in retrospect it was a stupid decision, because OLAP databases are not suited for low latency querying. However it <em>can</em> work with a generalized OLAP database and e.g. Snowflake or Clickhouse would manage this use case better. The reason it didn't work though, was that p99 latency was off the charts, p50 was in the 200ms range, which is slow but acceptable for a dashboarding use-case. However p99 was 2-10 seconds! This completely destroys any UX, especially the randomness of it. The way I understand it is that the latency comes from the Leader. The Leader receives the query, parses it and then <em>compiles</em> a C binary that it then ships to the executor nodes who then run it. </p>
<p>Secondly using Redshift to back BI tooling can work if the size of the data and the size of the team is small. However at a certain point you have to start trading usability for cost, because either you scale up and down, taking the cluster offline for 15 min at a time, or you end up running it at close to the max spec at times when you don't need it. Additionally, because only the executor nodes can scale, the leader is always going to be the bottleneck, especially with use-cases like dbt, when many metadata queries are executed. I don't honestly know if this problem is better on Serverless, but in our testing it was worse and more expensive. But we also haven't spent that much time trying to optimize it.</p>
<p>Therefore I don't really understand the point of Redshift, I think the only way I can see it working is if all the ETL is done <em>before</em> Redshift and then stored on S3 and loaded into Redshift, which is then only used for querying on ready tables. I would avoid using it as a general purpose data warehouse and I would avoid using it if latency is of concern.</p>
<h2 id="thougts-on-snowflake">Thougts on Snowflake</h2>
<p>Snowflake is a different beast altogether. It has the ability to spin up and down the compute power within seconds (or faster). So the second concern from Redshift goes away. You can simply create a warehouse allocation for different use-cases (e.g. different parts of the org, different projects, different tools) and give a correctly sized warehouse to each. When the warehouse is not needed, it automatically goes away and you stop paying for it.</p>
<p>The storage costs are higher than S3 but overall negligible, they <em>usually</em> come out to about 10% of the bill.</p>
<p>The warehouses can scale <em>up</em> and <em>out</em>. Up, means that you get bigger machines and out means you get more of them. So Up can take <em>bigger</em> queries and Out can take <em>more</em> queries in parallel.</p>
<p>With respect to user-facing features, there are 3 categories where Snowflake has a significant upper hand over Redshift:</p>
<ol>
<li>UI and Apps</li>
<li>Data Governance</li>
<li>Data Sharing</li>
</ol>
<h3 id="ui-and-apps">UI and Apps</h3>
<p>Overall, Snowflake has a nice UI called Snowsight that has some rudimentary text editor where the users can write SQL queries and execute them. It even has some basic charting capability so you can create dashboards directly in Snowsight. The one thing I would like to still see is Vim key bindings in the editor.</p>
<p>On top of that there are some interestng new options coming up, such as Streamlit, which is a way to write interactive web apps that work with data in Snowflake using Python.</p>
<p>On the admin side, there are also nice amenities for monitoring the spending and load of the warehouses. Also the query plan visualizer is worth mentioning.</p>
<h3 id="data-governance">Data Governance</h3>
<p>Snowflake has 2 interesting features here. (1) Role hierarchies: Users assume roles and the roles can inherit their privileges from other roles. But at the end of the day, all the queries are logged into the audit log with the user that was logged in as the source. (2) Data masking: this is an amazing feature that lets you assign "tags" (actually these should be called something else since it's a key-value pair) and then based on these tags, you can mask out the data in certain columns or tables from a group of people based on their assumed roles. This let's you set up a rule that e.g. Marketing can see the users table, but can't see the users' physical addresses, while analysts can not see the PII at all. Importantly, the masking is done when the data is queried, so it will look differend depending on who's looking. You can also apply different UDFs that do the masking, e.g. you can just hash the value being masked, so you still can do things like <code>count(distinct <col>)</code> over the masked data.</p>
<h3 id="data-sharing">Data Sharing</h3>
<p>This is a very powerful idea, because it lets 3rd parties make applications or data sets and make them easily available to you. Same goed the other way around. If you want to sell your data, you can do so through the data marketplace. From the snowflake perspective, all you see is that there is a new database you can read data from. And the kicker is that dbt, even though normally it can only connect to one data source, doesn't mind! You just need to explicitly specify the database in your query!</p>
<h2 id="conclusion">Conclusion</h2>
<p>I think by this point it's clear which way I'm leaning personally. I think if you're on AWS, Snowflake is still probably the best option for any mid-size and above company. It ended up being about 40% cheaper and our ETL ended up being 70% faster on Snowflake compared to Redshift and on top we got better UI and better data governance!</p>
Merging configurations with Nix2023-11-03T00:00:00+00:002023-11-03T00:00:00+00:00https://mirosval.sk/blog/2023/nix-merging-configs/<h1 id="merging-configurations-with-nix">Merging configurations with Nix</h1>
<p>Here's a problem: How to split up my Nix config?</p>
<p>I would like to split my config in multiple files, but:</p>
<ol>
<li>I want the individual files to be as stand-alone as possible, so I can copy them between projects, or give to friends</li>
<li>I want to edit system config from those stand-alone units</li>
</ol>
<p>This is a real problem without Nix that is usually dealt with by using something like Bash <code>>></code> operator to append text to the end of a config file and hope for the best. Or if your config files are YAML, you have to resort to <code>jq</code> or <code>regex</code>-fu.</p>
<p>How does Nix help us?</p>
<p>Lets imagine a scenario where I have a config like this:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#65737e;"># configuration.nix
</span><span style="color:#8fa1b3;">{ </span><span>pkgs, ... </span><span style="color:#8fa1b3;">}</span><span>:
</span><span>{
</span><span> </span><span style="color:#65737e;"># Some service definitions
</span><span> </span><span style="color:#d08770;">services</span><span>.</span><span style="color:#d08770;">abc </span><span>= { ... };
</span><span> </span><span style="color:#d08770;">services</span><span>.</span><span style="color:#d08770;">xyz </span><span>= { ... };
</span><span>
</span><span> </span><span style="color:#65737e;"># Networking
</span><span> </span><span style="color:#d08770;">networking</span><span>.</span><span style="color:#d08770;">firewall </span><span>= {
</span><span> </span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">allowedTCPPorts </span><span>= [
</span><span> </span><span style="color:#d08770;">3000 </span><span style="color:#65737e;"># Service ABC
</span><span> </span><span style="color:#d08770;">3001 </span><span style="color:#65737e;"># Service XYZ
</span><span> ];
</span><span> };
</span><span>}
</span></code></pre>
<p>Now I would like to split this by extracting services into their own files:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#65737e;"># configuration.nix
</span><span style="color:#8fa1b3;">{ </span><span>pkgs, ... </span><span style="color:#8fa1b3;">}</span><span>:
</span><span>{
</span><span> </span><span style="color:#d08770;">imports </span><span>= [
</span><span> </span><span style="color:#a3be8c;">./services/abc/default.nix
</span><span> </span><span style="color:#a3be8c;">./services/xyz/default.nix
</span><span> ];
</span><span>
</span><span> </span><span style="color:#65737e;"># Networking
</span><span> </span><span style="color:#d08770;">networking</span><span>.</span><span style="color:#d08770;">firewall </span><span>= {
</span><span> </span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> };
</span><span>}
</span></code></pre>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#65737e;"># services/abc/default.nix
</span><span style="color:#8fa1b3;">{ </span><span>pkgs, ... </span><span style="color:#8fa1b3;">}</span><span>:
</span><span>{
</span><span> </span><span style="color:#d08770;">services</span><span>.</span><span style="color:#d08770;">abc </span><span>= { ... };
</span><span>
</span><span> </span><span style="color:#d08770;">networking</span><span>.</span><span style="color:#d08770;">firewall</span><span>.</span><span style="color:#d08770;">allowedTCPPorts </span><span>= [
</span><span> </span><span style="color:#d08770;">3000 </span><span style="color:#65737e;"># Service ABC
</span><span> ];
</span><span>}
</span></code></pre>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#65737e;"># services/xyz/default.nix
</span><span style="color:#8fa1b3;">{ </span><span>pkgs, ... </span><span style="color:#8fa1b3;">}</span><span>:
</span><span>{
</span><span> </span><span style="color:#d08770;">services</span><span>.</span><span style="color:#d08770;">xyz </span><span>= { ... };
</span><span>
</span><span> </span><span style="color:#d08770;">networking</span><span>.</span><span style="color:#d08770;">firewall</span><span>.</span><span style="color:#d08770;">allowedTCPPorts </span><span>= [
</span><span> </span><span style="color:#d08770;">3001 </span><span style="color:#65737e;"># Service XYZ
</span><span> ];
</span><span>}
</span></code></pre>
<p>This is very nice because now I can create my separate modules and Nix will take care of merging the configurations, in this case by concatenating the <code>networking.firewall.allowedTCPPorts</code> list from the two modules. This option is defined <a href="https://github.com/NixOS/nixpkgs/blob/fa804edfb7869c9fb230e174182a8a1a7e512c40/nixos/modules/services/networking/firewall.nix#L13-L22">here</a> and the magical bit is that, by default, lists are merged using concatenation and additionally the <code>apply</code> function <a href="https://github.com/NixOS/nixpkgs/blob/fa804edfb7869c9fb230e174182a8a1a7e512c40/nixos/modules/services/networking/firewall.nix#L9-L10"><code>canonicalizePortList</code></a> is applied to make sure each port only appears once.</p>
<p>The documentation for this is <a href="https://nixos.org/manual/nixos/stable/#sec-option-declarations-eot">Extensible Option Types</a> and basically what happens is that modules options are declared using types in such a way that also defines how these are supposed to be merged. </p>
Getting Started with Nix2023-10-27T00:00:00+00:002023-10-27T00:00:00+00:00https://mirosval.sk/blog/2023/nix-getting-started/<h1 id="getting-started-with-nix">Getting started with Nix</h1>
<p>I have been using Nix for more than 6 months now and I have gradually migrated 5 boxes to it. I'm by no means a Nix expert, but at this point I feel like I can write down my experience for others that are just starting out. Before it becomes second nature and I forget all the pain I went through to get here. There is never enough getting started tutorials!</p>
<blockquote>
<p>Note: I'm using Flakes exclusively, they are listed as an experimental tech, but are sufficiently stable and well supported.</p>
</blockquote>
<h2 id="who-is-this-for">Who is this for</h2>
<p>I think Nix is great, however it will be a lot easier to learn if you have some Functional Programming chops. I'm writing this tutorial for developers, as they can get the biggest lift out of Nix. For others, should you learn Nix? I don't know, probably <a href="https://github.com/hlissner/dotfiles#frequently-asked-questions">not</a>.</p>
<h2 id="what-is-nix">What is Nix</h2>
<p>The term is a bit confusing, because it means multiple things at once. But in short there are:</p>
<ul>
<li>Nix Language</li>
<li>Nixpkgs the package repository</li>
<li>Nix CLI</li>
<li>Nix OS</li>
</ul>
<p>The most important thing to know is that you can use any subset of these and still get some benefits of Nix. See [What to use Nix for](#What to use nix for) for some examples. Xe Iaso has aa nice image explaining some of these relationships on <a href="https://xeiaso.net/talks/asg-2023-nixos/">their blog</a>.</p>
<h3 id="installing-nix">Installing Nix</h3>
<p>There is the <a href="https://nixos.org/download#download-nix">official way</a>, but it's probably better to start using the <a href="https://github.com/DeterminateSystems/nix-installer">Determinate Systems' Nix Installer</a>, mainly because it enables flakes by default and it's a lot easier to uninstall if you want to do that:</p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
</span></code></pre>
<h3 id="nix-language">Nix Language</h3>
<p>The language is used across Nix to write all the instructions and descriptions of packages and how to install them. But a simple way to think of it is like a language that makes JSONs.</p>
<p>It does not look like it, but its a fully-fledged functional programming language that skews towards map-like data structures. The language relies heavily on a data structure called <code>attrset</code> or sometimes just <code>set</code>, which is somewhat similar to JSON or e.g. Python's <code>dict</code>, but has some really nice quality of life properties on top.</p>
<p>Probably the best place to learn the Nix Language is <a href="https://nixos.org/manual/nix/stable/language/">The Manual</a>, but it is written like an explanation of the <a href="https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form">BNF</a> so if you're not familiar with that, there is a very nice one-pager Nix tutorial <a href="https://github.com/tazjin/nix-1p">Nix 1P</a>.</p>
<h3 id="nixpkgs">Nixpkgs</h3>
<p>The gateway to Nixpkgs is its <a href="https://search.nixos.org/packages?">Search Page</a>. From there you can find all the packages that are available for easy use within the Nix ecosystem, but being a build system, if your package is not present, getting it installed through Nix is probably easier than you think.</p>
<p>When you find your package on Nixpkgs, there are helpful links. One to the package Homepage, which is usually the GitHub repo, but the other called <code>Source</code> goes to <a href="https://github.com/NixOS/nixpkgs/blob/nixos-23.05/pkgs/tools/text/sd/default.nix#L29">NixOS/nixpkgs on GitHub</a>. This is a magical place where all the packages are defined and its a great source of examples if you're trying to build your own packages.</p>
<h3 id="nix-cli">Nix CLI</h3>
<p>There are 2 kinds of commands, the legacy (e.g. <code>nix-shell</code>) and the new Flake-based (e.g. <code>nix flake</code>, <code>nix develop</code>, etc.).</p>
<ul>
<li><code>nix --show-trace <command></code> is useful when you need more debugging information</li>
<li><code>nix run <app></code> will run a binary, downloading and building it if necessary, e.g. <code>nix run deadnix</code></li>
<li><code>nix flake check</code> will evaluate the nix expression in <code>flake.nix</code> and check all the assertions and checks. This can help you find issues without even building anything</li>
<li><code>nix flake update</code> will update the <code>flake.lock</code> with the newest versions of inputs</li>
<li><code>nix flake lock --update-input nixpkgs</code> will update <code>flake.lock</code> with the newest version of <code>nixpkgs</code> input</li>
</ul>
<h3 id="nixos">NixOS</h3>
<p>I'm using NixOS on 2 Raspberries and a Mini PC headless. This is where the Nix Option search comes in handy the most, all of the OS things that can be configured have an Option and they are pretty well documented. </p>
<p>I don't have much of a reference point with other Linux distros that I would use regularly, but I think its already a testament to Nix that it got me excited about Linux again and I began investing in PC hardware.</p>
<h2 id="why-use-nix">Why use Nix</h2>
<p>There are many reasons, including the ones mentioned on the <a href="https://nixos.org/">Nix Homepage</a>, like Reproducible, Declarative and Reliable, but for me the main reason is that, after some initial hurdles, it's very <em>pleasant</em> to work with. It just feels good to make some changes to my flake and do a <code>make switch</code> and it goes ahead and applies them. And I know that if it worked once, there is a very good chance it will continue working.</p>
<p>In the past I've set up many computers, from my laptops to VPSes and it's always a crushing feeling to have to start over with something you've already done because the OS upgrade messed up. I don't like that!</p>
<p>I guess that feeling comes down to the Declarative bit. The configuration is written as a set of files and it's only those files that you need to re-create the box the same way it was, if you accidentally mess up.</p>
<h2 id="what-to-use-nix-for">What to use Nix for</h2>
<p>Here is where I think most Nix tutorials are lacking. When I started with Nix, I was struggling to understand which parts work together and how and what are the kind of things I can do with Nix. So here I'm going to try to summarize the kind of use cases I have found useful with Nix.</p>
<h3 id="dev-environments">Dev environments</h3>
<p>This can be done without the need for <em>any</em> other of the below use cases. This is hard to stress enough! The basic steps are:</p>
<ol>
<li>Install Nix</li>
<li>Set your repo up with a flake</li>
<li>Done</li>
</ol>
<p>You can just use nix to get a reproducible environment in a single repo you have checked out locally and nothing else. It does not interfere with the rest of the system, you can confinue using your fav OS, your favourite package manager (Brew, or whatever, or even no package manager).</p>
<p>When this is set up, all you do is <code>cd</code> into the project and you're in your reproducible fixed shell managed by Nix. Nix will replace your <code>$PATH</code> with a path to <code>/nix/store</code> where your project's dependencies are installed, so whatever you do to your system outside the repo has no effect inside the repo folder.</p>
<p>Commit the flake and all your colleagues who have Nix set up and check out the repo will have the exact same software installed on their computers. </p>
<p>When you have multiple repos, each with their own Nix setup, Nix will maintain an union of all the necessary packages, so if 2 repos are using the same Python, it will only be installed once, but if they are using two different Python versions, each will have their own. It will only be active in that repo's directory.</p>
<p>Examples of how to use dev environments with flakes can be found in the <a href="https://github.com/mirosval/dotfiles/tree/e280568501bdf86ba34a1f1cb288a9a2dc4b5b43/templates">templates</a> of my dotfiles. I use these when starting a new project like this:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#bf616a;">nix flake init </span><span>-</span><span style="color:#bf616a;">t </span><span>"</span><span style="color:#a3be8c;">github:mirosval/dotfiles?dir=templates#rust</span><span>"
</span></code></pre>
<p>I can't remember that so I have a <a href="https://github.com/mirosval/dotfiles/blob/e280568501bdf86ba34a1f1cb288a9a2dc4b5b43/home/navi/cheats/common.cheat#L76">cheat code</a>:</p>
<pre data-lang="text" style="background-color:#2b303b;color:#c0c5ce;" class="language-text "><code class="language-text" data-lang="text"><span># Initialize new nix project from flake template
</span><span>nix flake --show-trace init -t "github:mirosval/dotfiles?dir=templates#<template>"
</span><span>
</span><span>$ template: ls -1 -F $HOME/.dotfiles/templates | grep '/$' | sed -e 's|/$||' | sort
</span></code></pre>
<h3 id="home">Home</h3>
<p>You can manage your home config (dotfiles) using Nix. This is mainly interesting if you are on MacOS or on Linux and don't want to switch to NixOS. It is a step up from just using Nix for dev environments. Now you are using that same <code>/nix/store</code> as before, but <code>$PATH</code> is modified globally for your user, not just in the repositories that are using Nix. </p>
<p>You can now use Nix to create and manage your dotfiles for you instead of using a tool like <a href="https://www.chezmoi.io/">Chezmoi</a> or managing them yourself using shell scripts (which is what I did prior to Nix). This is nice, because you visit and apply your entire configuration every time you do a <code>switch</code>. No more install scripts that only run the first time you're setting up your machine and need maintenance every time you need them. Conversely it will also remove files that are no longer linked. So if you remove a program that had an entry in <code>~/.config</code> that was linked to the <code>/nix/store</code>, that entry is also removed.</p>
<p>Additionally this allows you to define a set of packages that should be installed and are available on your <code>$PATH</code>. For example if you are using <a href="https://neovim.io/">Neovim</a>, you can get rid of your Neovim package manager and instead use Nix to install your plugins. If you've ever ran <code>:PlugUpgrade</code> or <code>:PackerSync</code> and ended up with a broken Neovim, because of a breaking change in one of your plugins with no way back to the previous state, do I have good news for you! With Nix you can simply switch to the previous generation to <em>immediately</em> have a working system again and you can deal with the breakage at the time of your choosing.</p>
<p>The component that enables this is called <a href="https://github.com/nix-community/home-manager">Home Manager</a>, it has a <a href="https://nix-community.github.io/home-manager/">manual</a> and <a href="https://mipmip.github.io/home-manager-option-search/">option search</a>.</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">description </span><span>= "</span><span style="color:#a3be8c;">Miro's dotfiles</span><span>";
</span><span>
</span><span> </span><span style="color:#d08770;">inputs </span><span>= {
</span><span> </span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nixos/nixpkgs/nixpkgs-unstable</span><span>";
</span><span> </span><span style="color:#d08770;">home-manager </span><span>= {
</span><span> </span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nix-community/home-manager</span><span>";
</span><span> </span><span style="color:#d08770;">inputs</span><span>.</span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">follows </span><span>= "</span><span style="color:#a3be8c;">nixpkgs</span><span>";
</span><span> };
</span><span> };
</span><span>
</span><span> </span><span style="color:#d08770;">outputs </span><span>= </span><span style="color:#8fa1b3;">{ </span><span>nixpkgs, home-manager, ... </span><span style="color:#8fa1b3;">}</span><span>:
</span><span> </span><span style="color:#b48ead;">let
</span><span> </span><span style="color:#d08770;">system </span><span>= "</span><span style="color:#a3be8c;">aarch64-darwin</span><span>";
</span><span> </span><span style="color:#d08770;">pkgs </span><span>= </span><span style="color:#96b5b4;">import </span><span style="color:#bf616a;">nixpkgs </span><span>{ </span><span style="color:#b48ead;">inherit </span><span style="color:#d08770;">system</span><span>; };
</span><span> </span><span style="color:#b48ead;">in
</span><span> {
</span><span> </span><span style="color:#d08770;">homeConfigurations</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="color:#bf616a;">home-manager</span><span>.</span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">homeManagerConfiguration </span><span>{
</span><span> </span><span style="color:#d08770;">modules </span><span>= [
</span><span> (</span><span style="color:#8fa1b3;">{ </span><span>lib, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#d08770;">programs</span><span>.</span><span style="color:#d08770;">home-manager</span><span>.</span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> })
</span><span> ];
</span><span> };
</span><span> };
</span><span>}
</span></code></pre>
<h3 id="local-computer">Local Computer</h3>
<p>Another level on top of Home Manager is managing the whole computer using Nix. I'm not sure how this works on Linux, because there I've always skipped directly to NixOS, but in theory this is where Nix would cooperate with your distro to manage higher level settings.</p>
<p>On MacOS, the tool to do this is called <a href="http://daiderd.com/nix-darwin/">nix-darwin</a>. It can maage system-level things, like Users, MacOS-specific settings, but also you can use it to install software from <a href="https://brew.sh/">Brew</a> or the Mac App Store.</p>
<p>On Linux, this is where we get to NixOS. Its a full operating system that you can install "traditionally" using an <a href="https://nixos.org/download.html">ISO</a> or a USB stick. But there are also some crazy ways, like <a href="https://github.com/elitak/nixos-infect">Nixos-infect</a> which will convert an existing Linux installation into a NixOS installation. This is especially interesting if you want to run Nix on a VPS and your VPS provider does not offer NixOS as one of the options.</p>
<p>A minimal nix-darwin setup looks like this:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">description </span><span>= "</span><span style="color:#a3be8c;">Miro's dotfiles</span><span>";
</span><span>
</span><span> </span><span style="color:#d08770;">inputs </span><span>= {
</span><span> </span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nixos/nixpkgs/nixpkgs-23.05-darwin</span><span>";
</span><span> </span><span style="color:#d08770;">home-manager </span><span>= {
</span><span> </span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nix-community/home-manager/release-23.05</span><span>";
</span><span> </span><span style="color:#d08770;">inputs</span><span>.</span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">follows </span><span>= "</span><span style="color:#a3be8c;">nixpkgs</span><span>";
</span><span> };
</span><span> </span><span style="color:#d08770;">darwin </span><span>= {
</span><span> </span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:lnl7/nix-darwin</span><span>";
</span><span> </span><span style="color:#d08770;">inputs</span><span>.</span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">follows </span><span>= "</span><span style="color:#a3be8c;">nixpkgs</span><span>";
</span><span> };
</span><span> };
</span><span>
</span><span> </span><span style="color:#d08770;">outputs </span><span>= inputs@</span><span style="color:#8fa1b3;">{ </span><span>self, nixpkgs, home-manager, darwin, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#d08770;">darwinConfigurations</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="color:#bf616a;">darwin</span><span>.</span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">darwinSystem </span><span>{
</span><span> </span><span style="color:#65737e;"># If you need to support more systems, use https://github.com/numtide/flake-utils
</span><span> </span><span style="color:#d08770;">system </span><span>= "</span><span style="color:#a3be8c;">aarch64-darwin</span><span>";
</span><span> </span><span style="color:#d08770;">modules </span><span>= [
</span><span> </span><span style="color:#a3be8c;">./hosts/mirosval/default.nix </span><span style="color:#65737e;"># <-- Your nix-darwin config goes into this file
</span><span> </span><span style="color:#bf616a;">home-manager</span><span>.</span><span style="color:#bf616a;">darwinModules</span><span>.</span><span style="color:#bf616a;">home-manager
</span><span> {
</span><span> </span><span style="color:#d08770;">nixpkgs </span><span>= </span><span style="color:#bf616a;">nixpkgs</span><span>;
</span><span> </span><span style="color:#d08770;">users</span><span>.</span><span style="color:#d08770;">users</span><span>."</span><span style="color:#a3be8c;">mirosval</span><span>".</span><span style="color:#d08770;">home </span><span>= "</span><span style="color:#a3be8c;">/Users/mirosval</span><span>";
</span><span> </span><span style="color:#d08770;">home-manager</span><span>.</span><span style="color:#d08770;">useGlobalPkgs </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">home-manager</span><span>.</span><span style="color:#d08770;">users</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="color:#bf616a;">home-manager</span><span>.</span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">homeManagerConfiguration </span><span>{
</span><span> </span><span style="color:#d08770;">modules </span><span>= [
</span><span> (</span><span style="color:#8fa1b3;">{ </span><span>lib, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#65737e;"># Your Home-manager config goes here
</span><span> </span><span style="color:#d08770;">programs</span><span>.</span><span style="color:#d08770;">home-manager</span><span>.</span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> })
</span><span> ];
</span><span> };
</span><span> } </span><span style="color:#65737e;"># No comma or colon here because this is the end of a List element
</span><span> ];
</span><span> };
</span><span> };
</span><span>}
</span></code></pre>
<h3 id="remote-computer">Remote Computer</h3>
<p>This is where my knowledge ends for the time being, I know that there are tools like <a href="https://github.com/NixOS/nixops">NixOps</a>, <a href="https://github.com/DBCDK/morph">Morph</a>, <a href="https://github.com/zhaofengli/colmena">Colmena</a> or <a href="https://github.com/serokell/deploy-rs">deploy-rs</a>. It is possible to manage remote nix installs with these but I have no experience with any of them yet.</p>
<p>I have some experience cross-building Raspberry PI images with NixOS. The basic workflow is to first create a builder, which is a docker image with NixOS running on it. You can use Docker's ability to switch target platforms to pretend that this is a computer on a different architecture. This is needed so that you can cross build raspberry <code>aarch64-linux</code> on <code>aarch64-darwin</code> or <code>x86_64-linux</code>. Then you instruct the builder to create the whole NixOS install and package it into an SD-card image. Now you can take this image to your Raspberry and you have NixOS on Raspberry!</p>
<h3 id="other-usecases">Other usecases</h3>
<p>I have seen others do these but I don't have experience with them myself</p>
<h4 id="ci">CI</h4>
<p>Using Nix on the CI is pretty straightforward, there is a <a href="https://github.com/marketplace/actions/the-determinate-nix-installer">Nix Installer GitHub Action</a>. There is also <a href="https://www.cachix.org/">Cachix</a> to speed up your builds.</p>
<h4 id="docker-images">Docker Images</h4>
<p>You can get Nix to build Docker images for you, this fits well, because an ideal Docker image only has the application and the minimum dependencies needed to run the application. Which is not the case if you build using the standard approach, especially not if you work with scripting languages like Python.</p>
<h2 id="how-to-go-about-learning-nix">How to go about learning Nix</h2>
<p>The key is to realize that Nix language is just a fancy way to write a single <code>attrset</code> (think a single big JSON). This is a <em>description</em> of what you would like to happen. Then various different Nix tools or commands take that description and do something with it. </p>
<p>With that in mind, there are two main questions you want to answer when working with nix:</p>
<p>(1.) What to write in my Nix file to create the right description</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">outputs </span><span>= _: {
</span><span> </span><span style="color:#d08770;">darwinConfigurations</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="background-color:#bf616a;color:#2b303b;">???</span><span>; </span><span style="color:#65737e;"># What goes here???
</span><span> };
</span><span>}
</span></code></pre>
<p>In other words, what is the schema? This confused me a lot in the beginning, because in theory you can return anything as a part of your <code>attrset</code>, however if you want to achieve specific things, you need to return specifically formatted <code>attrset</code>s.</p>
<p>The first place this is documented is the <a href="https://nixos.wiki/wiki/Flakes#Output_schema">Flakes Nix Wiki</a>. That lists the top level keys you can use in a flake. But it's not super helpful, because:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span> </span><span style="color:#65737e;"># Used by `nix develop`
</span><span> </span><span style="color:#bf616a;">devShells</span><span>."</span><span style="color:#a3be8c;"><system></span><span>".</span><span style="color:#bf616a;">default </span><span style="background-color:#bf616a;color:#2b303b;">=</span><span> </span><span style="color:#96b5b4;">derivation</span><span style="background-color:#bf616a;color:#2b303b;">;</span><span>
</span></code></pre>
<p>is nice, but what's a <code>derivation</code>? It turns out that the correct answer there is <a href="http://ryantm.github.io/nixpkgs/builders/special/mkshell/"><code>mkShell</code></a> (yes, technically you can put other things there).</p>
<p>It also looks like Determinate Systems has put some effort into collecting these <a href="https://determinate.systems/posts/flake-schemas">schemas</a>.</p>
<p>(2.) What tool to use to do something with my description</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">outputs </span><span>= _: {
</span><span> </span><span style="color:#d08770;">darwinConfigurations</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="color:#bf616a;">_</span><span>;
</span><span> </span><span style="color:#65737e;"># ^ having this, what do I invoke on the command line?
</span><span> };
</span><span>}
</span></code></pre>
<p>Or what to run to get it to work. The confusion there is related to the Flake split, but I think, generally speaking commands of the form <code>nix-*</code> are legacy and <code>nix *</code> are Flake-based. So for instance <code>nix flake check</code> will validate the assertions that are specified in a Flake and report any errors.</p>
<p>So specifically, for a nixos configuration:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">outputs </span><span>= _: {
</span><span> </span><span style="color:#d08770;">nixosConfigurations</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="color:#bf616a;">_</span><span>;
</span><span> };
</span><span>}
</span></code></pre>
<p>you have to switch to it using:</p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>nixos-rebuild switch --show-trace --flake .#mirosval
</span></code></pre>
<p>For a nix-darwin configuration:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">outputs </span><span>= _: {
</span><span> </span><span style="color:#d08770;">darwinConfigurations</span><span>.</span><span style="color:#d08770;">mirosval </span><span>= </span><span style="color:#bf616a;">_</span><span>;
</span><span> };
</span><span>}
</span></code></pre>
<p>you have to switch to it using:</p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>nix build .#darwinConfigurations.mirosval.system
</span><span>result/sw/bin/darwin-rebuild switch --flake .#mirosval
</span></code></pre>
<h3 id="recommended-tooling">Recommended tooling</h3>
<p>I'm using <a href="https://github.com/oxalica/nil">Nil LSP</a> as my LSP server (with Neovim), additionally I've found <a href="https://github.com/NerdyPepper/statix">statix</a> to be very helpful in understanding Nix language patterns. If you have nix, you can run it simply with <code>nix run nixpkgs#statix check</code>. Similarly you can use <a href="https://github.com/astro/deadnix">deadnix</a> to find unused code: <code>nix run nixpkgs#deadnix</code>.</p>
<p>To find nix packages, I rely on the <a href="https://duckduckgo.com/bangs">DuckDuckGo Bangs</a>, so for example <code>!nixpkgs deadnix</code> goes to <a href="https://search.nixos.org/packages?channel=23.05&show=deadnix&from=0&size=50&sort=relevance&type=packages&query=deadnix">Nixpkgs search results for <code>deadnix</code></a>. You can use <code>!nixopt</code> to search through Nix options.</p>
<h3 id="common-errors">Common errors</h3>
<h4 id="forgetting-to-git-add-a-file">Forgetting to git add a file</h4>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>error: getting status of '/nix/store/mhni7gc5azcggzl4rqqd78w4i7clp43i-source/home/aaaa': No such file or directory
</span></code></pre>
<p>When you are adding files that you want to reference from your flake.nix, they need to be added to git using <code>git add <file></code> before the flake can be built. If you forget to do that you get an error like this.</p>
Using macvlan networking with Nixos containers2023-10-04T00:00:00+00:002023-10-04T00:00:00+00:00https://mirosval.sk/blog/2023/nix-macvlan-networking/<p>Problem Context</p>
<p>I would like to be able to put something like <code>https://dash</code> into the browser on my local network or Tailscale and have it load my homelab dashboard.</p>
<p>There are a couple of caveats though:</p>
<ul>
<li>It should work independently for my home network and Tailscale</li>
<li>It should also include DNS-based ad blocking</li>
</ul>
<p>Aspiration</p>
<p>I would like to be able to run a Nixos container in the macvlan mode, so that on my home network level I get an IP address I can work with. This is necessary, because DHCP can not advertise DNS on a port different from 53.</p>
<p>I would like my DNS solution to be able to be DRY, I want to rely on my router to resolve hostnames and I would just like to specify <code>CNAME</code>s for them in order to reach multiple services that are hosted on my home server.</p>
<p>Unexpected issues</p>
<p>Working with Nixos containers if very pleasant, its possible to configure the software that runs inside the container using Nix, so I don't need 2-phase builds, first the image then the container. Also, Nix is much nicer to work with than Dockerfiles.</p>
<p>However, after having configured macvlan for my container, I found out that the network is not coming up.</p>
<pre data-lang="sh" style="background-color:#2b303b;color:#c0c5ce;" class="language-sh "><code class="language-sh" data-lang="sh"><span style="color:#bf616a;">ifconfig -a
</span></code></pre>
<p>Turns out <a href="https://github.com/NixOS/nixpkgs/issues/91751">I'm not the only one</a>. So it turned out that the best way to fix this is to switch to <code>systemd.network</code>. After having done that, I can run <code>nixos-rebuild switch</code> and the network inside of the container is connected correctly.</p>
<p>My NixOS configuration looks like this:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#8fa1b3;">{ </span><span>config, lib, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#d08770;">systemd</span><span>.</span><span style="color:#d08770;">network </span><span>= {
</span><span> </span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">networks </span><span>= {
</span><span> "</span><span style="color:#a3be8c;">40-enp2s0</span><span>" = {
</span><span> </span><span style="color:#d08770;">matchConfig</span><span>.</span><span style="color:#d08770;">Name </span><span>= "</span><span style="color:#a3be8c;">enp2s0</span><span>";
</span><span> </span><span style="color:#d08770;">networkConfig</span><span>.</span><span style="color:#d08770;">DHCP </span><span>= "</span><span style="color:#a3be8c;">yes</span><span>";
</span><span> };
</span><span> };
</span><span> };
</span><span> </span><span style="color:#d08770;">networking </span><span>= {
</span><span> </span><span style="color:#d08770;">networkmanager</span><span>.</span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#d08770;">useNetworkd </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> };
</span><span>}
</span></code></pre>
<p>there is no bridge to the macvlan, so the connections from the host to the container have to go around the main interface, I'd like to fix that in the future.</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#8fa1b3;">{ </span><span>config, lib, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#d08770;">systemd</span><span>.</span><span style="color:#d08770;">network </span><span>= {
</span><span> </span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#65737e;"># silly fix for the service failing on nixos rebuild
</span><span> </span><span style="color:#d08770;">wait-online</span><span>.</span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">mkForce </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#d08770;">networks </span><span>= {
</span><span> "</span><span style="color:#a3be8c;">40-enp2s0</span><span>" = {
</span><span> </span><span style="color:#d08770;">matchConfig</span><span>.</span><span style="color:#d08770;">Name </span><span>= "</span><span style="color:#a3be8c;">enp2s0</span><span>";
</span><span> </span><span style="color:#d08770;">networkConfig</span><span>.</span><span style="color:#d08770;">DHCP </span><span>= "</span><span style="color:#a3be8c;">yes</span><span>";
</span><span> };
</span><span> };
</span><span> };
</span><span> </span><span style="color:#d08770;">networking </span><span>= {
</span><span> </span><span style="color:#d08770;">hostName </span><span>= "</span><span style="color:#a3be8c;">butters</span><span>"; </span><span style="color:#65737e;"># Define your hostname.
</span><span> </span><span style="color:#d08770;">networkmanager</span><span>.</span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#d08770;">useNetworkd </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">nameservers </span><span>= [
</span><span> "</span><span style="color:#a3be8c;">1.1.1.1</span><span>"
</span><span> "</span><span style="color:#a3be8c;">1.0.0.1</span><span>"
</span><span> ];
</span><span> };
</span><span>}
</span></code></pre>
<p>and the container side looks like this:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#8fa1b3;">{ </span><span>config, lib, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#d08770;">containers</span><span>.</span><span style="color:#d08770;">lan-dns </span><span>= {
</span><span> </span><span style="color:#d08770;">autoStart </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">ephemeral </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">macvlans </span><span>= [ "</span><span style="color:#a3be8c;">enp2s0</span><span>" ];
</span><span> </span><span style="color:#d08770;">privateNetwork </span><span>= </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#65737e;">#extraFlags = ["-U"]; # private user namespace
</span><span> </span><span style="color:#d08770;">config </span><span>= </span><span style="color:#8fa1b3;">{ </span><span>pkgs, ... </span><span style="color:#8fa1b3;">}</span><span>: {
</span><span> </span><span style="color:#d08770;">networking </span><span>= {
</span><span> </span><span style="color:#d08770;">useDHCP </span><span>= </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#d08770;">useNetworkd </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">useHostResolvConf </span><span>= </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#d08770;">firewall </span><span>= {
</span><span> </span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">interfaces</span><span>."</span><span style="color:#a3be8c;">mv-enp2s0</span><span>".</span><span style="color:#d08770;">allowedUDPPorts </span><span>= [ </span><span style="color:#d08770;">53 </span><span>];
</span><span> };
</span><span> };
</span><span> </span><span style="color:#d08770;">systemd</span><span>.</span><span style="color:#d08770;">network </span><span>= {
</span><span> </span><span style="color:#d08770;">enable </span><span>= </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#d08770;">networks </span><span>= {
</span><span> "</span><span style="color:#a3be8c;">40-mv-enp2s0</span><span>" = {
</span><span> </span><span style="color:#d08770;">matchConfig</span><span>.</span><span style="color:#d08770;">Name </span><span>= "</span><span style="color:#a3be8c;">mv-enp2s0</span><span>";
</span><span> </span><span style="color:#d08770;">address </span><span>= [
</span><span> "</span><span style="color:#a3be8c;">192.168.1.3/24</span><span>"
</span><span> ];
</span><span> </span><span style="color:#d08770;">networkConfig</span><span>.</span><span style="color:#d08770;">DHCP </span><span>= "</span><span style="color:#a3be8c;">yes</span><span>";
</span><span> </span><span style="color:#d08770;">dhcpV4Config</span><span>.</span><span style="color:#d08770;">ClientIdentifier </span><span>= "</span><span style="color:#a3be8c;">mac</span><span>";
</span><span> };
</span><span> };
</span><span> };
</span><span> };
</span><span> };
</span><span>}
</span></code></pre>
<p>This way the <code>lan-dns</code> container shows up on the outer network as a separate host and can serve DNS without problems.</p>
Slow dependency resolution in Poetry2023-05-19T00:00:00+00:002023-05-19T00:00:00+00:00https://mirosval.sk/blog/2023/poetry-slow-resolution/<p>Dependency resolution in packaging systems in general is a messy subject. </p>
<p>As a rule of thumb, it's good if libraries specify their dependencies as <em>loosely</em> as possible, so that the downstream libraries or applications that use them have flexibility to choose a version that will work with other dependencies.</p>
<p>Applications, however, probably want to specify their dependencies as <em>strictly</em> as possible, to ensure build reproducibility.</p>
<p>But this leads to an interesting constraint-solving problem. You have a set of dependencies for your application and for each dependency, you have a set of constraints. If the problem is over-constrained (i.e. library A requires library C version 1.0, but library B <em>also</em> requires library C, but version 2.0) the dependency resolution fails! If the problem is under-constrained, you have many valid versions to consider. And of these many versions, some might have their dependencies that need to fit with the rest of the application's dependencies.</p>
<p>To compound this problem, the dependency hierarchy forms a tree, and while <em>all</em> the dependencies must fit together, you're "discovering" the full set of dependencies only during resolution.</p>
<p>The dependency resolution is a hard problem and is in fact <a href="https://research.swtch.com/version-sat?ref=thefeedbackloop.xyz">NP Complete</a>.</p>
<p>So the thing barely works as-is and any delays along the hot path have the potential to become <em>very</em> slow for any sizeable project.</p>
<p>I have run into an example of this recently in Poetry. It is pretty well documented in their <a href="https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow">F.A.Q</a>.</p>
<p>Quote from Poetry FAQ:</p>
<blockquote>
<p>This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but to download the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process.</p>
</blockquote>
<p>The basic setup of the problem is:</p>
<ol>
<li>At the start of dependency resolution, not all dependencies are known yet</li>
<li>Some Python packages on PyPI don't fully declare their dependencies in the metadata</li>
</ol>
<p>This means that as the solver descends, it needs to download all of those packages to inspect <em>their</em> dependencies and this can be <em>very</em> slow.</p>
<p>An example with specific libraries:</p>
<ol>
<li>Specify <a href="https://github.com/aws/amazon-redshift-python-driver/blob/master/requirements.txt"><code>redshift-connector</code></a> as a dependency in your Python project</li>
<li>The <code>redshift-connector</code> specifies its dependency on <a href="https://github.com/boto/boto3"><code>boto3</code></a> as <code>boto3>=1.9.201,<2.0.0</code>. This is hundreds of valid versions, since <code>1.9.201</code> was released in August 2019, and <code>boto3</code> is released almost daily.</li>
<li>Poetry proceeds to download all of these versions to check them for their dependencies (I stopped it after 3h of doing this)</li>
</ol>
<p>Thankfully the solution is also mentioned in their FAQ:</p>
<blockquote>
<p>At the moment there is no way around it. However, if you notice that Poetry is downloading many versions of a single package, you can lessen the workload by constraining that one package in your pyproject.toml more narrowly. That way Poetry does not have to sift through so many versions of it, which may speed up the locking process considerably in some cases.</p>
</blockquote>
<p>So in our specific example, the solution is to add <code>boto3</code> as an explicit dependency, this will tell Poetry early on that there is only 1 version to try solving with.</p>
<p>To identify which (transitive) dependency is causing problems, you can run e.g.</p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>poetry update --vvv
</span></code></pre>
<p>Once you identify the dependency, add it explicitly to your <code>pyproject.toml</code>:</p>
<pre data-lang="toml" style="background-color:#2b303b;color:#c0c5ce;" class="language-toml "><code class="language-toml" data-lang="toml"><span style="color:#bf616a;">boto3 </span><span>= "</span><span style="color:#a3be8c;">1.26.137</span><span>"
</span></code></pre>
My tmux workflow2023-02-20T00:00:00+00:002023-02-20T00:00:00+00:00https://mirosval.sk/blog/2023/tmux-workflow/<h1 id="why-tmux">Why tmux</h1>
<p>From the user point of view a terminal is a simple command line where you first navigate into a directory, then run some commands there, navigate elsewhere and run some other commands. It can be tedious to move around all the time, so if you're using an app like the Terminal.app or iTerm2, you could open a few tabs, one per project you're working on. However then when you close the app, you'll have to set up all the locations again when you start working.</p>
<p>This is the workflow I use tmux for. Tmux <em>can</em> be used for lots of other things, like rich environments on remote machines with SSH tunneling, but today I want to focus on the local workflow.</p>
<p>There are some important concepts in tmux:</p>
<ul>
<li><code>sessions</code> are sets of windows, you can switch between them using <code>ctrl + b s</code></li>
<li><code>windows</code> are sets of panes arranged in a specific way. You can switch between them using <code>ctrl + b <num></code> where <code><num></code> is the number of the window you want to select</li>
<li><code>panes</code> which are the splits of a window. Generally you can have more than one pane displayed at a time. You switch between panes using <code>ctrl + b <left/right/top/bottom arrow></code></li>
</ul>
<h1 id="tmux-setup">Tmux setup</h1>
<p>To get started, you can install tmux using the simple <code>brew install tmux</code>. Once you have it installed, you can simply start it by typing </p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>tmux
</span></code></pre>
<p>This will start a shell that looks almost identical to the one you had before. Only now you have some keyboard shortcuts to start with.</p>
<p>Tmux has this concept of prefix, by default it's <code>ctrl + b</code>. This means that every tmux keyboard shortcut is prefixed with <code>ctrl + b</code> (meaning you first press and hold <code>ctrl</code>, then press <code>b</code>, then release both keys. Then you press the shortcut key).</p>
<p><img src="https://mirosval.sk/blog/2023/tmux-workflow/screenshot.png" alt="tmux screenshot" /></p>
<h1 id="tmuxp">tmuxp</h1>
<p>Now <a href="https://github.com/tmux-python/tmuxp">tmuxp</a> is a program that lets you manage tmux sessions declaratively. You can install it using</p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>brew install tmuxp
</span></code></pre>
<p>Then in <code>~/.config/tmuxp/dot.yaml</code> you can have a file that looks like this:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">session_name</span><span>: </span><span style="color:#a3be8c;">dotfiles
</span><span style="color:#bf616a;">windows</span><span>:
</span><span> - </span><span style="color:#bf616a;">window_name</span><span>: </span><span style="color:#a3be8c;">dotfiles
</span><span> </span><span style="color:#bf616a;">layout</span><span>: </span><span style="color:#a3be8c;">main-vertical
</span><span> </span><span style="color:#bf616a;">options</span><span>:
</span><span> </span><span style="color:#bf616a;">main-pane-width</span><span>: </span><span style="color:#a3be8c;">50%
</span><span> </span><span style="color:#bf616a;">shell_command_before</span><span>: </span><span style="color:#a3be8c;">cd ~/.dotfiles
</span><span> </span><span style="color:#bf616a;">panes</span><span>:
</span><span> - </span><span style="color:#a3be8c;">nvim
</span><span> - </span><span style="color:#a3be8c;">blank
</span><span> - </span><span style="color:#bf616a;">window_name</span><span>: </span><span style="color:#a3be8c;">blog
</span><span> </span><span style="color:#bf616a;">layout</span><span>: </span><span style="color:#a3be8c;">main-vertical
</span><span> </span><span style="color:#bf616a;">options</span><span>:
</span><span> </span><span style="color:#bf616a;">main-pane-width</span><span>: </span><span style="color:#a3be8c;">50%
</span><span> </span><span style="color:#bf616a;">shell_command_before</span><span>: </span><span style="color:#a3be8c;">cd ~/blog
</span><span> </span><span style="color:#bf616a;">panes</span><span>:
</span><span> - </span><span style="color:#a3be8c;">nvim
</span><span> - </span><span style="color:#a3be8c;">blank
</span></code></pre>
<p>What this configures is that there will be a session called <code>dotfiles</code>, in it there will be 2 windows, one called <code>dotfiles</code> and one called <code>blog</code>. Each of these windows starts in its own directory and consists of 2 panes: neovim and blank pane. The <code>main-vertical</code> layout means that the first split will be vertical, i.e. you have the main pane on the left half of the screen and the right half of the screen will be split horizontally into multiple panes if you add more than the 2.</p>
<p>Its a good idea to logically group projects into these sessions. For example I have one session for the ETL, one for infrastructure and another for services.</p>
<p>In order to go from empty terminal to a loaded project you can write</p>
<pre data-lang="shell" style="background-color:#2b303b;color:#c0c5ce;" class="language-shell "><code class="language-shell" data-lang="shell"><span>tmuxp load ~/.config/tmux/dot.yaml
</span></code></pre>
<p>It will set up all the sessions and windows and seed them with the right commands so you can jump right into work.</p>
<p>Tmux, nvim and others are usually super light-weight, so I just keep them all open at the same time. Thanks to sessions and windows I can have order in all my projects and it barely makes a dent in my computer's RAM. Which can not be said about most IDEs.</p>
Everything is a migration2023-01-14T00:00:00+00:002023-01-14T00:00:00+00:00https://mirosval.sk/blog/2023/everything-is-a-migration/<p>When you are starting with Software Engineering right now, there is really a lot of material already available online, from videos, books, tutorials to online classes. Most of this material is either structured to teach you how a certain concept works or how a specific technology works. When you're starting your first job, you think you'll start on a green-field new project, choose the tech stack and go to town.</p>
<p>But then you join a company and they already have some legacy application, nobody knows how it really works but it supports 80% of the business. </p>
<p>And so:</p>
<blockquote>
<p>Any change to an existing application is a migration. </p>
</blockquote>
<p>My guesstimate is that 80% of the time of a software engineer that is not spent in meetings is spent on shepherding some long-running migration. So it makes sense to think about how to get better at migrations as a means of getting better as a software engineer.</p>
<h2 id="what-is-a-migration">What is a migration</h2>
<p><em>A migration is the process of applying a change on a system.</em></p>
<p>Lets unpack this, <em>process</em> implies this is going to take a while, <em>applying</em> means that <strong>you</strong> have to do it, <em>a change</em> signifies that the new system is different from the old one and <em>the system</em> can really be anything from company processes, through organization to software.</p>
<p>Some examples:</p>
<p>Introducing an RFC process at the company is a migration that consists of preparing the process definitions and ceremonies, getting initial approvals from management, presenting it to everyone and finally following up and making sure its adhered to.</p>
<p>Replacing a legacy service with a service written in Rust while maintaining the API compatibility can look something like writing the new service, deploying it with a dark traffic split, checking all is working and finally switching the client traffic to the new service.</p>
<p>Migrating a data warehouse means first rewriting all your SQL in a format that is compatible with both warehouses, bringing the new warehouse online, checking all the data and calculations and finally switching all the users to the new warehouse.</p>
<h2 id="why-everything-is-a-migration">Why everything is a migration</h2>
<p>Even the most mundane tasks, like updating a README on a repo are migrations. The PR merge triggers CI/CD and Kubernetes will perform a rolling update of the service. This is a migration that's already been automated, so you don't experience it directly, but its still good to know what happens when there is a failure in these processes.</p>
<p>A migration consists of 3 steps:</p>
<ol>
<li>Pre-refactoring</li>
<li>Change</li>
<li>Post-refactoring</li>
</ol>
<p>First you have to prepare the old system for the change. Most commonly this is some kind of clean up that is necessary to make the change atomic later. It can literally be a refactoring, but it can mean bringing up a whole new service, or setting up a new warehouse.</p>
<p>Then you can finally do the switch. The deployment or switch the load balancer or present the RFC process.</p>
<p>After the change, if everything went well, you likely still need to clean up. Maybe you had to introduce some constructs to make the migration easy or now that you're on the new system, some old ways can be replaced with new ones.</p>
<h2 id="what-makes-migrations-hard">What makes migrations hard</h2>
<p>Often there is an easy "migration" when system uptime is not required. For example when you develop locally and you mess up your database, you can just nuke it and start from the beginning. Not a problem because nobody else depended on that database to be available.</p>
<p>In production however, the most common source of complications is the requirement to keep the system running without downtime.</p>
<p>This usually means that migrations are done by bringing the changed system online and then atomically, reversibly switching to it. </p>
<h2 id="how-to-get-better-at-migrating">How to get better at migrating</h2>
<p>The most important thing to keep in mind is that things can and often <em>do</em> go wrong during a migration. What differentiates good migrations from bad migrations is how easy it is to get back to a working system when something went wrong.</p>
<p>So to get better at migrations, these are the things to think about:</p>
<ol>
<li>Get better at finding <em>what</em> can go wrong (make sure you have your known unknowns covered)</li>
<li>Prepare scenarios for each of the potential problems on how to restore to a working state</li>
<li>Make sure all your migration steps are <em>atomic</em>, meaning they can be applied and reversed with a single change</li>
</ol>
Better IDEA Keymap2016-07-30T00:00:00+00:002016-07-30T00:00:00+00:00https://mirosval.sk/blog/2016/idea-settings/<h2 id="boring-ramblings-skip">Boring ramblings (skip)</h2>
<p>First of all I really want to shout out to Intellij for building such awesome tools.
I used Eclipse, Netbeans, Visual Studio but they all suck tremendously. OK, Eclipse
and Netbeans are OSS, so you can live with that. But VS? Its almost cruel how shitty
it is. It is so shitty that Intellij made a plugin to fix it. With the plugin its good.
But costs > $1000 for professional use. Its shameful, really.</p>
<p>IDEA is fantastic, it starts quickly, doesn't freeze up, offers all these advanced
features, code completions, rewrites and so on. I really recommend it to everyone.</p>
<p>And it has a community version which is free and paid version is well worth its money.
I pay monthly subscription for my PyCharm.</p>
<p>So why would I ramble about it in the first place? IDEA has not one but 3 shortcut sets:</p>
<ul>
<li>Mac OS X 10.5+ (⌃R for Run) - Not very macOS-like</li>
<li>Mac OS X (F10 for Run) - WTF?</li>
<li>Eclipse (Mac OS X) (⌘⇧F11 for Run) - ?????</li>
</ul>
<p>These are just some examples, but generally the keymaps are this stupid overall.
I genuinely wonder who came up with these. F keys on macOS have been traditionally
used for media keys, screen dimming and such. I do not want to use Fn modifier for
that, and I sure as hell will not type Fn⌘⇧F11 just to Run my code, which is probably
the most common action I need a shortcut for. Stupid.</p>
<h2 id="guiding-principles">Guiding Principles</h2>
<p>I wanted my shortcuts to be as usable as possible, so I tried to keep to these rules:</p>
<ul>
<li>One handed shortcuts for the most common actions</li>
<li>Two handed only if it is inevitable</li>
<li>Ideally Left hand, since most people will use the mouse with the right</li>
<li>Do not interfere with existing macOS shortcuts</li>
<li>Try to use popular shortcuts from other editors (Atom, Sublime)</li>
</ul>
<h2 id="disclaimers-and-limitations">Disclaimers and limitations</h2>
<ul>
<li>I mainly work with Scala and Python, so the shortcuts are optimized for IDEA
and PyCharm, but I'm sure they can be used in other Intellij products.</li>
<li>The multiple selection shortcut could annoy some people, because ⌘Click is
mapped to documentation or something like that by default</li>
</ul>
<h2 id="show-me-the-shortcuts">Show me the shortcuts!</h2>
<p>Multiple cursors:</p>
<ul>
<li>⌘Click to place an additional cursor</li>
<li>⌘D to select the next occurrence of a selected word (same as in Atom/Sublime Text)</li>
<li>Esc to cancel selection</li>
</ul>
<p>Running tests:</p>
<ul>
<li>Assuming you are using a testing framework like ScalaTest</li>
<li>Place cursor into the body of the test function/method/describe/it block</li>
<li>If this is the first time you’re running this test (or if you have run a different test previously) start with:
<ul>
<li>⌘⇧R to run the test</li>
<li>⌘⇧T to run the test with debugger</li>
</ul>
</li>
<li>If this is already the second time you’re running this same test/suite:
<ul>
<li>⌘R to run the test</li>
<li>⌘T to run the test with debugger</li>
</ul>
</li>
<li>⌘E while stopped in the debugger</li>
<li>⌘. to stop run/test/other running process</li>
<li>⌘[ to Step Over in the Debugger</li>
<li>⌘] to Step Into in the Debugger</li>
<li>⌘\ to Step Out in the Debugger</li>
</ul>
<p>Navigation Around IDE:</p>
<ul>
<li>⌘P Search Symbols</li>
<li>⌘⇧P Search Classes</li>
<li>⌘B Jump to definition</li>
</ul>
<p>Navigation In the Editor:</p>
<ul>
<li>⌥→ and ⌥← To move one word at a time</li>
<li>⌥⌃→ and ⌥⌃← To move one word at a time, respecting CamelHumps</li>
<li>⌥⇧→ and ⌥⇧← To select one word at a time</li>
<li>⌥⇧⌃→ and ⌥⇧⌃← To select one word at a time, respecting CamelHumps</li>
<li>⌥⌫ Delete Line</li>
</ul>
<p>Other shortcuts:</p>
<ul>
<li>⌘5 to Make/Build project, this is useful if you need to see if your project compiles</li>
<li>⇧⇧ Search Everything</li>
<li>⌘⇧A Search in actions</li>
<li>⌘, Preferences</li>
</ul>
<h2 id="download">Download</h2>
<ul>
<li><a href="settings.jar">My Entire IDEA Settings JAR</a></li>
<li><a href="keymaps-only.jar">Keymaps Only JAR</a></li>
</ul>
<h2 id="share">Share</h2>
<p>Please take these shortcuts, use them, improve on them, share the changes. Maybe
we can convince Intellij to adopt them for the next IDEA Release.</p>
Lamp Project: Printed Circuit Boards2015-03-19T00:00:00+00:002015-03-19T00:00:00+00:00https://mirosval.sk/blog/2015/lamp-project-4/<ol>
<li><a href="https://mirosval.sk/blog/2015/lamp-project/">Project Overview</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-2/">Hardware</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-3/">Arduino Software</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-4/">Printed Circuit Boards</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-5/">Assembly Timelapse</a></li>
</ol>
<h2 id="design-tool">Design Tool</h2>
<p>I haven't really searched all that far to discover <a href="http://www.cadsoftusa.com/">Eagle CAD</a>, it seems to be probably the most widely used software for PCB design. I have watched two tutorial videos on YouTube by Jeremy Blum, <a href="https://www.youtube.com/watch?v=1AXwjZoyNno">Schematic Design</a> and <a href="https://www.youtube.com/watch?v=CCTs0mNXY24">Printed Circuit Board Layout</a>. That gave me a pretty good idea about how to use the software. I won't pretend there weren't any issues, but all were successfully resolved with a search on StackOverflow. Overall I think it is a very good tool. I've never used it before but I have been able to do all I wanted from it in maybe an hour.</p>
<h2 id="schematic-design">Schematic Design</h2>
<p>I started out with a schematic design. I already had a working prototype of the lamp, so it wasn't so hard to use that to start things off. Overall the schematic is divided into three sub-circuits from the left:</p>
<ul>
<li>Power, voltage dropdown for the RFduino</li>
<li>The RFduino</li>
<li>Darlington Array to increase the voltage back</li>
</ul>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-4/schematic.png" alt="Board Schematic" /></p>
<p>The most problematic part of the voltage regulator was to actually obtain the part I chose: <a href="http://www.ti.com/product/lp2950-33">LP2950-33</a>. Ideally you would choose a LP2950-30 here, as RFduino's reference voltage is 3.0V, but it is rated to up to 3.6V, so 3.3V is fine. I have, by accident supplied it with more than that (think ~5V) and it has surprisingly survived.</p>
<p>I have used two 10μF capacitors, one at input and one at output to further stabilize the voltage. </p>
<p>The RFduino is pretty self-explanatory, I have hooked the VCC and GND pins and I've connected the GPIO pins 2, 3 and 4 to the input of the darlington array.</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-4/darlington.png" alt="Darlington Array" /></p>
<p>The last part was the <a href="https://www.sparkfun.com/datasheets/IC/ULN2003A.pdf">Darlington Array</a>. If you take a look at the schematic below, you'll see how it's wired inside. The LED strip has one positive wire and then three ground wires, each for one of the RGB colors. The positive end is connected directly to VCC, 1B-3B are connected to the output of the RFduino, 1C-3C are connected to the LED strip and the ground (not shown in the little schematic) is connected to the circuit ground.</p>
<p>One of the biggest issues I had was to find the parts, Eagle has a large database of components, but still, things like RFduino are not there. Also I needed to make sure I can eventually buy the things somewhere.</p>
<h2 id="board-design">Board Design</h2>
<p>After finishing the schematic, I proceeded with the board design. First I have tried to automatically layout the components, but Eagle has made a complete mess of things, so in the end I had to lay out everyting myself anyway. The process can get a bit awkward with the bends of the wires, but if it does, you can just delete the wire and do it again. You can set the widths of the wires, so I've chosen slightly thicker wires for the 12V lines and slightly thinner for the rest. I've also managed to keep all the connections to the one side of the board, which just makes it look better in my opinion.</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-4/board.png" alt="Board Design" /></p>
<h2 id="manufacturing">Manufacturing</h2>
<p>Somewhere on the internet I have found <a href="https://oshpark.com/">OSH Park</a> a company from USA that will make PCBs from Eagle CAD files: 3 copies of the board for $5 per square inch (imperial, really??) and ship it anywhere in the world for free. My board cost me under $20, I have ordered it on Feb 26, it got shipped on Mar 6 and arrived in Prague on Mar 18. This manufacturing delay makes mistakes super expensive, but luckily the board works as designed.</p>
<h2 id="assembly">Assembly</h2>
<p>I have assembled the components onto the board today, it took me about an hour and was a really pleasant experience. And the fact that it has worked on the first try was really satisfying. Here have a look at the finished product.</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-4/board-finished.jpg" alt="Board Finished" />
<img src="https://mirosval.sk/blog/2015/lamp-project-4/finished.jpg" alt="Finished Lamp" /></p>
<h2 id="final-notes">Final Notes</h2>
<p>OSH Park has good <a href="https://oshpark.com/guidelines">guidelines</a> to follow (they also have a file that you can import to Eagle that will check your design). I suggest reading those before submitting anything.</p>
<p>The boards that have arrived look stunning, but there are a couple of things I noticed that could be improved:</p>
<ul>
<li>Some joints don't have the small metal area surrounding the hole, which makes them slightly more difficult to solder (see the holes for RFduino and the voltage regulator)</li>
<li>The other joints have the metal area around holes on both sides of the board, even though only one side has wires</li>
<li>The boards had some rough edges around the places where they were broken apart from the panel that had to be sanded, but this is just a minor issue</li>
</ul>
<p>Most of these are my fault, I will try to fix them next time I have boards printed.</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-4/board-defects.jpg" alt="Board Defects" />
<img src="https://mirosval.sk/blog/2015/lamp-project-4/soldering-regulator.jpg" alt="Board Design" /></p>
Lamp Project: Arduino Software2015-03-12T00:00:00+00:002015-03-12T00:00:00+00:00https://mirosval.sk/blog/2015/lamp-project-3/<ol>
<li><a href="https://mirosval.sk/blog/2015/lamp-project/">Project Overview</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-2/">Hardware</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-3/">Arduino Software</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-4/">Printed Circuit Boards</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-5/">Assembly Timelapse</a></li>
</ol>
<p>Below you can find the whole source code for the Arduino sketch uploaded to RFduino. It's pretty simple for now, it contains mappings to the pins that output <a href="http://en.wikipedia.org/wiki/Pulse-width_modulation">PWM</a> for the LEDs. It also contains definitions of 2 modes that are supported: <em>MODE_RESET</em> and <em>MODE_HOLD</em>. The reset mode will turn of the light on disconnect, while the hold mode will keep the color even if the remote disconnects. It can then be changed if someone send it a new color.</p>
<p>The most interesting I guess is the onReceive function that is called by the RFduino stack when it has received new data. It receives a pointer to the data array and its length. This also defines the lamp protocol, so the data format is as follows:</p>
<ul>
<li>One data packet is 4 bytes long</li>
<li>Position 0 is the mode, currently only values 0 and 1 are supported, 0 will reset color after the client disconnects, 1 will keep the color until changed</li>
<li>Position 1-3 are RGB values to be written to the output</li>
</ul>
<p>The mode is finally resolved in the RFduinoBLE_onDisconnect function.</p>
<pre data-lang="c" style="background-color:#2b303b;color:#c0c5ce;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#b48ead;">#include </span><span><</span><span style="color:#a3be8c;">RFduinoBLE.h</span><span>>
</span><span>
</span><span style="color:#65737e;">// Lamp Modes
</span><span style="color:#b48ead;">const</span><span> byte MODE_RESET = </span><span style="color:#d08770;">0 </span><span><< </span><span style="color:#d08770;">0</span><span>;
</span><span style="color:#b48ead;">const</span><span> byte MODE_HOLD = </span><span style="color:#d08770;">1 </span><span><< </span><span style="color:#d08770;">0</span><span>;
</span><span>
</span><span style="color:#65737e;">// pin 3 on the RGB shield is the red led
</span><span style="color:#65737e;">// (can be turned on/off from the iPhone app)
</span><span style="color:#b48ead;">int</span><span> led_r = </span><span style="color:#d08770;">3</span><span>;
</span><span style="color:#b48ead;">int</span><span> led_g = </span><span style="color:#d08770;">4</span><span>;
</span><span style="color:#b48ead;">int</span><span> led_b = </span><span style="color:#d08770;">2</span><span>;
</span><span>
</span><span>byte mode = </span><span style="color:#d08770;">0</span><span>;
</span><span>
</span><span style="color:#b48ead;">void </span><span style="color:#8fa1b3;">setup</span><span>() {
</span><span> </span><span style="color:#65737e;">// led turned on/off from the iPhone app
</span><span> </span><span style="color:#bf616a;">pinMode</span><span>(led_r, OUTPUT);
</span><span> </span><span style="color:#bf616a;">pinMode</span><span>(led_g, OUTPUT);
</span><span> </span><span style="color:#bf616a;">pinMode</span><span>(led_b, OUTPUT);
</span><span>
</span><span> RFduinoBLE.</span><span style="color:#bf616a;">advertisementData </span><span>= "</span><span style="color:#a3be8c;">mlamp</span><span>";
</span><span>
</span><span> </span><span style="color:#65737e;">// start the BLE stack
</span><span> RFduinoBLE.</span><span style="color:#bf616a;">begin</span><span>();
</span><span>}
</span><span>
</span><span style="color:#b48ead;">void </span><span style="color:#8fa1b3;">loop</span><span>() {
</span><span>
</span><span>}
</span><span>
</span><span style="color:#b48ead;">void </span><span style="color:#8fa1b3;">RFduinoBLE_onDisconnect</span><span>()
</span><span>{
</span><span> </span><span style="color:#b48ead;">if</span><span>(mode & MODE_HOLD) {
</span><span> Serial.</span><span style="color:#bf616a;">println</span><span>("</span><span style="color:#a3be8c;">Mode was set to MODE_HOLD, so will leave LEDs on</span><span>");
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> </span><span style="color:#65737e;">// MODE_RESET
</span><span> Serial.</span><span style="color:#bf616a;">println</span><span>("</span><span style="color:#a3be8c;">Mode was set to MODE_RESET, so turning off LEDs</span><span>");
</span><span> </span><span style="color:#65737e;">// don't leave the led on if they disconnect
</span><span> </span><span style="color:#bf616a;">analogWrite</span><span>(led_r, </span><span style="color:#d08770;">0</span><span>);
</span><span> </span><span style="color:#bf616a;">analogWrite</span><span>(led_g, </span><span style="color:#d08770;">0</span><span>);
</span><span> </span><span style="color:#bf616a;">analogWrite</span><span>(led_b, </span><span style="color:#d08770;">0</span><span>);
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">void </span><span style="color:#8fa1b3;">RFduinoBLE_onReceive</span><span>(</span><span style="color:#b48ead;">char </span><span>*</span><span style="color:#bf616a;">data</span><span>, </span><span style="color:#b48ead;">int </span><span style="color:#bf616a;">len</span><span>)
</span><span>{
</span><span> </span><span style="color:#b48ead;">if</span><span>(len != </span><span style="color:#d08770;">4</span><span>) {
</span><span> </span><span style="color:#b48ead;">return</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// 2nd byte is mode
</span><span> mode = (byte)data[</span><span style="color:#d08770;">0</span><span>];
</span><span>
</span><span> </span><span style="color:#65737e;">// 3rd, 4th, 5th are r,g,b
</span><span> byte r = data[</span><span style="color:#d08770;">1</span><span>];
</span><span> byte g = data[</span><span style="color:#d08770;">2</span><span>];
</span><span> byte b = data[</span><span style="color:#d08770;">3</span><span>];
</span><span>
</span><span> </span><span style="color:#65737e;">// set the color
</span><span> </span><span style="color:#bf616a;">analogWrite</span><span>(led_r, r);
</span><span> </span><span style="color:#bf616a;">analogWrite</span><span>(led_g, g);
</span><span> </span><span style="color:#bf616a;">analogWrite</span><span>(led_b, b);
</span><span>}
</span></code></pre>
Lamp Project: The Hardware Prototype2015-03-07T00:00:00+00:002015-03-07T00:00:00+00:00https://mirosval.sk/blog/2015/lamp-project-2/<ol>
<li><a href="https://mirosval.sk/blog/2015/lamp-project/">Project Overview</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-2/">Hardware</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-3/">Arduino Software</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-4/">Printed Circuit Boards</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-5/">Assembly Timelapse</a></li>
</ol>
<p>I have, obviously, started with an idea. I didn't like any of the exisiting smart lamp solutions, mostly because they lacked customizability. So my thought process was to start with some minimal requirements. I have come up with the following:</p>
<ul>
<li>RGB LED Strip for the actual light</li>
<li>Bluetooth for controls</li>
<li>Some sort of Arduino to process it all</li>
</ul>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-2/finished-board.jpg" alt="Finished prototype board" /></p>
<h2 id="choosing-the-rgb-led-strip">Choosing the RGB LED Strip</h2>
<p>There are lots of different options, there are single-color strips, RGB strips or <a href="http://www.adafruit.com/products/1138">programmable strips</a>, where you can change the color of every single LED. I though I would be fine with just controlling the color of the entire strip so I went with something like <a href="http://www.ebay.com/itm/5M-5050-RGB-SMD-LED-Waterproof-Flexible-Strip-300-LEDs-44-Key-IR-Remote-/180992529478">this</a>.</p>
<p>I don't need the stupid IR controller, but I found that sometimes you can get the same thing <em>with</em> the controller for less than without it. The key characteristics are:</p>
<ul>
<li>SMD 5050 - this represents the size of a single LED</li>
<li>60 LED/m - I want this to be bright</li>
<li>12V DC - operating voltage</li>
<li>1A/m - required current per 1 meter of the strip</li>
</ul>
<p>These things are really key, because everything else in the system is based on these values. If you change anything, be sure to adjust the rest of the system accordingly.</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-2/detail-led-strip.jpg" alt="LED Strip Detail" /></p>
<h2 id="next-arduino">Next, Arduino</h2>
<p>I have come across <a href="http://www.rfduino.com/product/rfd22102-rfduino-dip/index.html">RFduino</a>. I have bought some basic kit when they were on Kickstarter, but I have since bought 6 more units. The trouble with this board is a bit in its availability. <a href="https://www.sparkfun.com/categories/274">Sparkfun</a> has it, but in the EU it is much more difficult to get your hands on it. I ended up buying from <a href="http://cz.mouser.com/new/rfdigital/rf-digital-rfduino/">Mouser</a>, but they are primarily a wholesale operation, so it's expensive to buy small quantities from them.</p>
<p>Having said that, I'm really happy with RFduino, it is programmable with the Arduino IDE, has nice sample code and iOS libraries, uses Bluetooth 4.0 Low Energy, which is super nice and, most importantly, I have found the connectivity <em>great</em>, it just always works. It has also survived a couple of blunders on my part, such as connecting it to 5V power instead of the recommended 3.3V.</p>
<h2 id="power-supply">Power supply</h2>
<p>So my calculation went like this: I want a 1m long LED strip controlled by RFduino, that is 12V * 1A = 12W. So I need a 12W power supply. I ended up buying <a href="http://www.gmelectronic.com/power-ac-adapter-12v-1500ma-connector-2-1mm-mw-p751-193">this one</a> because it has enough power, nice connector and is relatively cheap.</p>
<h2 id="connecting-it-all-together">Connecting it all together</h2>
<p>Now I have a LED strip and a power supply which both operate at 12V, but I have a RFduino which operates at 3.3V. So the problem is twofold:</p>
<ol>
<li>Convert 12V to 3.3V to power the RFduino</li>
<li>Drive the 12V LED with the signal from RFduino</li>
</ol>
<p>This in the end translates into 2 subsystems that need to be designed. A power regulator circuit to power RFduino from 12V and an amplifier or a relay that could drive the LEDs.</p>
<h3 id="voltage-regulator-for-the-rfduino-12v-to-3v">Voltage Regulator for the RFduino 12V to 3V</h3>
<p>This was the hardest part to figure out for me. I didn't want anything overly complicated and in the end I've settled for the <a href="http://www.ti.com/product/lp2950-33">LP2950-33</a> (<a href="http://www.onsemi.com/pub_link/Collateral/LP2950-D.PDF">Datasheet</a>). It looks like a transistor, has 3 legs, one for input, one for output and one for ground. The usual usage example I found involved capacitors between output and ground and optionally between input and ground to further stabilize the voltage. LP2950 comes in different varietes, so I chose the one that had it's output voltage closest to RFduino's 3V.</p>
<h3 id="darlington-array-to-drive-the-led-strip">Darlington Array to drive the LED Strip</h3>
<p>The last thing to do was to bring the PWM pin output from RFduino into the strip. The way the strip is connected is that there is a +12V cable and three different grounds, one for each of the RGB channels. So basically what the darlington does, is connect each of the RGB channels to the ground such, that the amount of current flowing through them can be controlled by RFduino.</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-2/finished-wiring.jpg" alt="Finished prototype board - wiring" /></p>
<h2 id="the-prototype">The Prototype</h2>
<p>I have built the prototype using an universal board with headers bought from <a href="http://www.aliexpress.com/snapshot/6468939297.html?orderId=65605779553704">Aliexpress</a> so that I can plug the RFduino in and out as I please. This is what it looks like when I turn it on:</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project-2/finished-on.jpg" alt="Finished prototype turned on" /></p>
Lamp Project: Assembly Timelapse2015-01-18T00:00:00+00:002015-01-18T00:00:00+00:00https://mirosval.sk/blog/2015/lamp-project-5/<ol>
<li><a href="https://mirosval.sk/blog/2015/lamp-project/">Project Overview</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-2/">Hardware</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-3/">Arduino Software</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-4/">Printed Circuit Boards</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-5/">Assembly Timelapse</a></li>
</ol>
<iframe width="560" height="315" src="https://www.youtube.com/embed/nxq5sJD4_Y8" frameborder="0" allowfullscreen></iframe>
Lamp Project2015-01-18T00:00:00+00:002015-01-18T00:00:00+00:00https://mirosval.sk/blog/2015/lamp-project/<ol>
<li><a href="https://mirosval.sk/blog/2015/lamp-project/">Project Overview</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-2/">Hardware</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-3/">Arduino Software</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-4/">Printed Circuit Boards</a></li>
<li><a href="https://mirosval.sk/blog/2015/lamp-project-5/">Assembly Timelapse</a></li>
</ol>
<p>I've had this idea for a project for quite some time now, but now I've actually come around to some real progress. The project specificaton is simple: I wanted to build an RGB LED lamp that could be controlled wirelessly. That idea in and of itself is not particularly new or unprecedented, however when I was looking at ready-made solutions online, I couldn't find anything that would work for me. I really just wanted something that could be programmed or controlled remotely. </p>
<h2 id="the-vision">The Vision</h2>
<p>I really just want a smart lamp, that would be able to do things like change color and intensity of light according to various outside stimuli, such as direct control via a mobile phone, automatic lighting up and dimming according to my sleep cycle, audio input, temperature and so on. I would like to be able to control multiple lamps from one place and optionally have "groups" of lamps that could be controlled separately. I could not find anything that would be both inexpensive and flexible enough for my programming needs (most of these cheap RGB LED strips come with an IR remote control, but that is a joke). I thought I could build something that would work for me from the following components:</p>
<ul>
<li>IKEA Lamp</li>
<li>Arduino</li>
<li>RGB LED Strip</li>
<li>Power supply</li>
<li>iOS Device</li>
<li>Raspberry PI</li>
</ul>
<p>##The current state of things</p>
<p><img src="https://mirosval.sk/blog/2015/lamp-project/lamp-1.jpg" alt="Current state of the RGB LED Lamp" /></p>
<p>I've built a lamp using an Arduino and a Bluetooth chip that could be controlled by Bluetooth 2.0 back in 2012. This was good, but not good enough. The RGB strip worked well, but Bluetooth connection was unreliable and prone to problems (unable to find the device, unable to connect, etc.)</p>
<p>In the summer of 2014 I have built another prototype based on the <a href="http://www.rfduino.com/">Rfduino</a>, the new hot platform that includes Arduino and Bluetooth 4.0 (or BLE) in one package. This prototype works flawlessly, the connection is stable and easily established, which is a huge plus.</p>
<p>I currently have 1 Rfduino-based lamp with a 1m RGB LED strip attached to it. The LED strip I'm currently using is something like <a href="http://www.amazon.co.uk/Waterproof-300LEDs-Flexible-Lighting-Decoration/dp/B009ZOLW04/ref=sr_1_46?s=lighting&ie=UTF8&qid=1421617842&sr=1-46&keywords=5050+rgb+led+strip">this</a>. The key parameters are 12V, ~1A/m, 5050 LEDs 60 pieces per meter. The cool part is that the strip can be cut every 5cm (every 3 LEDs). So I have bought a 5m roll and I'm going to cut it into 5 1m long strips for use in 5 lamps.</p>
<p>The 1m LED strip I have draws 1.2A at 12V, which means it has a wattage of 14.4W. This determines all of the other components that are needed. For power supply I have used something like <a href="http://uk.farnell.com/powerpax/sw3526/ac-dc-power-supply-12v-1-25a-euro/dp/1971795">this</a>. At 15W it supplies enough power for the LED strip, Rfduino and the remaining circuitry. </p>
<p>I have designed a simple protocol that is able to send the RGB color to the lamp and is able to address up to 255 lamps. iOS is limited to 10 simmultaneous BT connections, so 255 is more than could be realistically adressed anyways. </p>
<p>And lastly I have built an iOS app that is able to set a solid color on a lamp, or alternatively can use the microphone to set the intensity of the light, while the user can pick the hue. I will discuss all these points in depth in future posts.</p>
<h2 id="immediate-future-of-the-project">Immediate future of the project</h2>
<p>Currently I have 1 lamp that works with the iOS app. I've recently received a shipment of 7 new Rfduinos and I've ordered more parts online which should arrive shortly. In the next weeks I'm going to build another prototype and test how the iOS app and the protocol cope with 2 or 3 lamps. I would also like to employ my Raspberry pi in the project, so that it could control the lamps without the need to use an iOS device. This would be perfect for things like morning alarm light, afternoon automatic lights and so on.</p>
<p>Another little subproject I have researched and would like to do is having PCBs printed. I have started designing the board using <a href="http://www.cadsoftusa.com/">Eagle</a>, but I have to wait until I can finalize the component list. I found an interesting place that could print my PCBs, <a href="https://oshpark.com/">OSHPark</a> and I would like to see how it works.</p>
<h2 id="long-term-ideas">Long term ideas</h2>
<p>There are a couple of things that I can see as a long term / low priority goals for the project:</p>
<ul>
<li>Have some sort of switch that would disentangle the lamp from the iOS controller</li>
<li>Have printed PCBs and a box with everything nicely contained inside</li>
</ul>
Roulette I (in Slovak)2014-07-10T00:00:00+00:002014-07-10T00:00:00+00:00https://mirosval.sk/blog/2014/roulette-1/<p>Tento článok pozostáva z 3 častí:</p>
<ol>
<li><a href="https://mirosval.sk/blog/2014/roulette-1/">Analýza pôvodného článku a metódy</a></li>
<li><a href="https://mirosval.sk/blog/2014/roulette-2/">Štatistická simulácia metódy a jej vyhodnotenie</a></li>
<li><a href="https://mirosval.sk/blog/2014/roulette-3/">O čo vlastne ide a závery</a></li>
</ol>
<hr />
<p>Na prvý pohľad je to zaujímavý článok od normálneho chlapíka. Ale po prvých pár paragrafoch úvodných kecov sa to rozbieha. Autor (ktorý nikde nie je podpísaný), píše 3 hlavné rozdiely medzi on-line a skutočným kasínom:</p>
<blockquote>
<p>Nikto nevidí, čo hráč robí pri počítači. Nie sú tam žiadni manažéri miestností, kamery alebo pozorní krupieri školení na sledovanie hráčskych metód a stratégií. Je tam iba hráč a jeho počítač.</p>
</blockquote>
<p>To je síce pravda, že pri online hraní vás nikto fyzicky nevidí, lebo ste doma. Avšak pán autor zabudol, spomenúť, že okrem vás a vášho počítača je tam ešte server, ktorý vlastní kasíno. Tento server vidí vaše karty, ako rýchlo, ako často, a akú sumu staviate, skrátka vidí oveľa viac ako vidí ktokoľvek vo fyzickom kasíne. V klasickom kasíne si nikto nezapisuje vaše všetky stávky.</p>
<blockquote>
<p>Nie sú žiadne limity na stávkovanie. V skutočnom kasíne musia hráči uzatvoriť stávky za menej ako 20 sekúnd. Ďalšou výhodou hrania on-line je, že hráči môžu pred uzavretím stávok počkať, takže môžu urobiť ten najlepší možný ťah.</p>
</blockquote>
<p>To, či v skutočnom kasíne limity sú alebo nie sú neviem, avšak nerozumiem ako vám pomôže čakať s uzatvorením stávky, keďže ruleta je prakticky hra jedného hráča a stávky ostatných hráčov nič nemenia.</p>
<blockquote>
<p>Keď hráte v on-line kasíne, môžete v skutočnosti dávať do stávky veľmi malé sumy, čo je nevyhnutné na to, aby táto metóda efektívne fungovala. V žiadnom skutočnom kasíne by vám to nedovolili.</p>
</blockquote>
<p>Znova neviem ako je to v skutočnom kasíne, ale to či táto metóda funguje, si ukážeme za chvíľu.</p>
<p>Pre lenivších tu opíšem celú tú metódu:</p>
<blockquote>
<p>Hlavný princíp tejto metódy je založený na opakovanom stávkovaní malých súm na základe prvej stávky. Používanie takejto schémy je 100 % legálne. Túto techniku som použil pri hraní rulety on-line a už v priebehu 5 mesiacov sa mi podarilo získať dosť peňazí na kúpu vysnívaného auta.</p>
<ol>
<li>Každú novú hru začínam stávkou 1 € na čierne číslo. </li>
<li>Ak prehrám (guľôčka padne na červené číslo), zdvojnásobím stávku na tú istú farbu (čiernu). </li>
<li>Ak vyhrám, začnem novú hru so stávkou 1 € na opačnú farbu.</li>
</ol>
</blockquote>
<p>Keď som si toto prečítal, začal som byť ešte podozrievavejší, pretože:</p>
<ul>
<li>"opakovanom stávkovaní malých súm na základe prvej stávky" - toto je tzv. <a href="http://en.wikipedia.org/wiki/Gambler's_fallacy">Gamblers Fallacy</a>, pomýlená predstava, že pravdepodobnosť ďalšieho výsledku závisí od predchádzajúceho výsledku. To jednoducho nie je pravda, to vie každý kto mal štatistiku. Jednotlívé ťahy sú na sebe nezávislé.</li>
<li>Kroky 2 a 3 menia farby. To je znova Gambler's fallacy, pretože, čierna a červená sú štatisticky ekvivalentné.</li>
<li>Hlavný princíp je stávkovanie malých súm, ale v kroku 2 zdvojnásobujeme stávku. To je exponenciálny rast. Aký to má dopad si ukážeme ďalej.</li>
</ul>
<blockquote>
<p>Niektorí ľudia si myslia, že je rozumnejšie začať s menšou sumou na konte a že to vykompenzujú tým, že budú hrať opatrne, ale pravdou je, že im hrozí oveľa vyššie riziko prehry.</p>
</blockquote>
<p>Nie, riziko prehry je v každej individuálnej stávke (ak sa bavíme o čiernej a červenej) rovnaké.</p>
<blockquote>
<p>98 % ľudí hrá bez akejkoľvek logiky alebo metódy. To je ten dav, na ktorom všetky kasína zarábajú svoje zisky. Títo ľudia nechápu hru z hľadiska kalkulácií a logických výsledkov.</p>
</blockquote>
<p>To je dosť možné. Ľudia, ktorí chápu ako ruleta funguje ju nehrajú a tie 2% sú ľudia, ktorí využívajú iné taktiky.</p>
<blockquote>
<p>nesmiete zabúdať na to, že všetci ostatní hráči majú iba 50 % šancu na výhru, ale ak zdvojnásobíte svoje stávky, vaša šanca na výhru sa bude maximálne približovať 100 %. A to je niečo, čo obyčajní hráči nikdy nedokážu pochopiť!</p>
</blockquote>
<p>Tu sú hneď 2 závažné chyby:</p>
<ol>
<li>Nielen ostatní hráči, <em>každý</em> má 50% šancu na výhru.</li>
<li>Nie je to 50%, ale 48,65% pretože v rulete je červená (18), čierna (18) a zelená (1) a teda červená a čierna majú šancu 18/37, ak sa bavíme o Francúszkej verzii. V Americkej je to ešte horšie, lebo tam sú 2 zelené polia. </li>
</ol>
<p>Graf, ktorý tam k tomu je je síce pekný, ale je to totálny nezmysel.</p>
<p>Na konci článku ešte nasleduje link na to spomínané on-line kasíno, kde sa dá takto skvele zarobiť a kopec komentárov od "úspešných" hráčov, ktorí takto zarobili už veľa peňazí.</p>
<p>Pokračujte <a href="https://mirosval.sk/blog/2014/roulette-2/">Štatistická simulácia metódy a jej vyhodnotenie</a></p>
Roulette II (in Slovak)2014-07-10T00:00:00+00:002014-07-10T00:00:00+00:00https://mirosval.sk/blog/2014/roulette-2/<p>Tento článok pozostáva z 3 častí:</p>
<ol>
<li><a href="https://mirosval.sk/blog/2014/roulette-1/">Analýza pôvodného článku a metódy</a></li>
<li><a href="https://mirosval.sk/blog/2014/roulette-2/">Štatistická simulácia metódy a jej vyhodnotenie</a></li>
<li><a href="https://mirosval.sk/blog/2014/roulette-3/">O čo vlastne ide a závery</a></li>
</ol>
<hr />
<p>Teraz sa pozrieme na simuláciu tejto metódy. Kompletný zdrojový kód nájdete ako <a href="http://nbviewer.ipython.org/gist/mirosval/dda218a0ae7cb1ab9449">IPython Notebook</a>.</p>
<p>Nasledujúca časť je však to podstatné. Implementácia logiky uvedenj metódy.</p>
<p>{% highlight python linenos %}
def play(tosses, account_balance):
# account_history podrzi vsetky hodnoty uctu pocas celej hry
account_history = [account_balance]
# pociatocna uroven stavky je 1 euro
bet = 1
# stavkujeme na cervenu alebo na ciernu, zaciname na ciernej
betting_on_red = False
# vyhrali sme toto kolo?
won = False</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span># iterujeme cez vsetky nahodne hody
</span><span>for num in tosses:
</span><span> # ak prave stavkujeme na cervenu a padla cervena, vyhravame
</span><span> if betting_on_red and num in red:
</span><span> won = True
</span><span> # ak stavkujeme na ciernu a padla cierna, tiez vyhravame
</span><span> elif not betting_on_red and num in black:
</span><span> won = True
</span><span> # inak sme prehrali
</span><span> else:
</span><span> won = False
</span><span>
</span><span> # ak sme toto kolo vyhrali
</span><span> if won:
</span><span> # pripiseme si vyhru
</span><span> account_balance += bet
</span><span> # zmenime farbu
</span><span> betting_on_red = not betting_on_red
</span><span> # resetujeme stavku na 1 euro
</span><span> bet = 1
</span><span> # ak sme prehrali
</span><span> else:
</span><span> # odpocitame si prehru
</span><span> account_balance -= bet
</span><span> # zvysime stavku na dvojnasobok
</span><span> bet *= 2
</span><span>
</span><span> # zapiseme si vysledok do historie
</span><span> account_history.append(account_balance)
</span><span>
</span><span> # ak sme klesli s uctom na alebo pod nulu, koncime
</span><span> if account_balance <= 0:
</span><span> return account_balance, account_history
</span><span>
</span><span># vratime vysledky
</span><span>return account_balance, account_history
</span></code></pre>
<p>{% endhighlight %}</p>
<p>Ak z tohoto kódu vynecháme riadky 56 a 57, môžeme dostať nasledujúci graf</p>
<p><img src="https://mirosval.sk/blog/2014/roulette-2/crash-arrow.jpg" alt="Crash" /></p>
<p>Tento graf ilustruje prečo je táto metóda problematická. Nestáva sa to vždy, ale občas sa stane, že nasleduje rovnaká farba veľa ťahov po sebe. V závislosti od počiatočného rozpočtu vždy existuje počet opakovaní ktorý je už likvidačný. Na úrovni 200€ je likvidačná 8x rovnaká farba zasebou.</p>
<table><thead><tr><th>Ťah</th><th style="text-align: right">Stávka</th><th style="text-align: right">Kumulatívna strata</th><th style="text-align: right">Stav účtu</th></tr></thead><tbody>
<tr><td>1</td><td style="text-align: right">1</td><td style="text-align: right">1</td><td style="text-align: right">199</td></tr>
<tr><td>2</td><td style="text-align: right">2</td><td style="text-align: right">3</td><td style="text-align: right">197</td></tr>
<tr><td>3</td><td style="text-align: right">4</td><td style="text-align: right">7</td><td style="text-align: right">193</td></tr>
<tr><td>4</td><td style="text-align: right">8</td><td style="text-align: right">15</td><td style="text-align: right">185</td></tr>
<tr><td>5</td><td style="text-align: right">16</td><td style="text-align: right">31</td><td style="text-align: right">169</td></tr>
<tr><td>6</td><td style="text-align: right">32</td><td style="text-align: right">63</td><td style="text-align: right">137</td></tr>
<tr><td>7</td><td style="text-align: right">64</td><td style="text-align: right">127</td><td style="text-align: right">73</td></tr>
<tr><td>8</td><td style="text-align: right">128</td><td style="text-align: right">255</td><td style="text-align: right">-55</td></tr>
</tbody></table>
<p>Pravdepodobnosť, že sa to stane je (19/37)^8 = 0,004835206373, čiže asi pol percenta. To môže vyzerať ako málo, ale treba si uvedomiť, že každý deň, podľa autora, hráme cca 180 hier!</p>
<p>Aby sme videli ako sa to prejaví v simulácii, pozrime sa na nasledujúci graf</p>
<p><img src="https://mirosval.sk/blog/2014/roulette-2/cumulative.png" alt="Cumulative" /></p>
<p>Tento graf ukazuje ako skončilo 1 000 hier, to sú cca 3 roky, 3 hodiny denne. Zelené krížiky sú hry ktoré skončili v pluse, červené, ktoré skončili v mínuse alebo na nule. Nula v tomto prípade znamená, že sme skončili s 200 eurami a za tie 3 hodiny sme nič nezarobili. Čiže oproti minimálnej mzde ste asi 9 eur v mínuse.</p>
<p>Ten istý program som pustil aj 10 000 krát, aby som získal presnejšie výsledky. Graf tu neukazujem, lebo je to len chaotickejšia verzia toho, ktorý tu už je. Zaujímavé sú však čísla. Minimálna výhra mi vyšla -455€, maximálna 111€ a pravdepodobnosť, že o všetko prídete cca na úrovni 34%. To znamená, že jedna tretina hier končí prinajlepšom na nule. Ale podstatný detail je, že nevyhráte 180€, ale len okolo 100€. Spolu to teda znamená, že v jednej tretine hier prerobíte 200€ a v dvoch tretinách zarobíte 100€, čo je dokopy cca 0€ keď sa to zráta a podčiarkne. To zhruba sedí s tým čo sa píše na <a href="http://en.wikipedia.org/wiki/Roulette">Wikipedii</a> v kolónke Expected value (on a $1 bet) (French), a síce, že očakávaná vyhra pri opakovanej stávke na jednu farbu je matematicky -$0.027.</p>
<p>V skratke, vyhráva kasíno. Ako vždy.</p>
<p>Pokračujte <a href="https://mirosval.sk/blog/2014/roulette-3/">O čo vlastne ide a závery</a></p>
Roulette III (in Slovak)2014-07-10T00:00:00+00:002014-07-10T00:00:00+00:00https://mirosval.sk/blog/2014/roulette-3/<p>Tento článok pozostáva z 3 častí:</p>
<ol>
<li><a href="https://mirosval.sk/blog/2014/roulette-1/">Analýza pôvodného článku a metódy</a></li>
<li><a href="https://mirosval.sk/blog/2014/roulette-2/">Štatistická simulácia metódy a jej vyhodnotenie</a></li>
<li><a href="https://mirosval.sk/blog/2014/roulette-3/">O čo vlastne ide a závery</a></li>
</ol>
<hr />
<p>Ako je teda možné, že tento chlapík zarobil "približne 6 tisíc mesačne"? Vysvetlení je niekoľko:</p>
<ul>
<li>Kasíno používa iný algoritmus na výpočet náhodných čísel. To je vysoko nepravdepodobné, pretože je to nelegálne. Náhodné čísla by mali byť uniformne rozložené.</li>
<li>Pán autor článku (neuviedol meno, tak neviem kto to je) je proste dieťa šťasteny a naozaj tie peniaze vyhral, ale nie preto, že jeho metóda je super, ale jednoducho mal šťastie</li>
<li>Pán autor nezarába na hraní v kasíne, ale na tom blogu. Ako? Referrals.</li>
</ul>
<p>Keď sa na ten blog bližšie pozrieme, vidíme, že je to vlastne len "blog" s <a href="http://braniblog.info/">1 stránkou</a> a doménu <a href="http://whois.domaintools.com/braniblog.info">vlastní niekto z Kyjeva</a>. </p>
<p>Všetky linky na stránke vedú na <a href="http://braniblog.info/2/eu.php">http://braniblog.info/2/eu.php</a>, ktorá nás presmeruje cez <a href="http://site.gotoeucasino.com/index.cgi?aname=mitkobombata1&zone_id=brask2">http://site.gotoeucasino.com/index.cgi?aname=mitkobombata1&zone_id=brask2</a> na <a href="http://slovakian.eucasino.com/">http://slovakian.eucasino.com/</a>, zaregistrované na Cypre, len tak mimochodom. Takže, keď sa cez tento link dostanete na kasíno, autor článku dostane nejaké percentá z vášho prvého vkladu, koľko presne to je alebo ako to funguje sa mi nepodarilo vypátrať.</p>
<p>Ešte pikoška na záver, spravil som si <a href="https://www.google.com/search?tbs=sbi:AMhZZivtxO7IivBaAiGAKuUK3NmymUxIq7xkPVqT2XM-AWfwAvNBSexSAO6JnaygNiXF6s47HtuX4LKqkC7DZtJGFUtED8WwlHRZ4TrcWU5AaG_1BOdPO9iCgXdhLvXIq-6KPjvMSHMkygzx7o8IfV3oY2W28SVHQm_1OZnsdwDLxDsN388rZpmq8_1Xy6HP0uPbrth_18MKWzD7XZHetHxvD26fUdrMRpKg6Y_1yJCDs5TT9CmByYPHtq_19Wd6zkAEeZiA6O6hKstt7foGhE-U5bpIPimLbYNA8OKUiUb3KDTt0vMIZMXTadEiAq_1hdqTSEfWP-N3iNYmLVwrz1vnJJn-Q1ArsCuWomb9-el0g-fBXr67z6J39h88tk1VZDBdzBgvVjzTiYe-R-HJBJlqrNXRJAJIeCwJDWp3dR3kqHkQG6kXk9Jt0FS0JOXMbyeIVfDu3TXRwQ3u9Hl35c5aa2FP8_1Nsha2fx0FoeAImWc5BKQgg-kDdgHR-Dyy9KLof66kFEJj253kU0Rx308o4F02u2kOEMFCXCx3NVhC0i3Eu7b0tnO0VQl56Kndao1uxQ6WdxliDiS3g9cFeGYcxuMoL1rHFmwdTS9F0GmSfrb6wh4NAUDSOjJuM_1V6VD6CaXKBn34a-BLCJaX3dNoUhAYYcWIr7GH_1lSKrxkUxJvfBJmBkCHyLbJVrMLz0bXMz4OyJjfBmYnlqDN3Fa7Yo2_1Romt81p7CSeR8VDObOVk_17EYsHIvk3mFFRfxgSDuJBFmMGYhrE6fuHPcCsJUDF15g0G3SmRRIIZkN9QS6JtN7NHnosAm_1LHk29ZG6m_1RiPqAhur1ggYq08ELh3Y_1WamMu2F-4emnu2kv1qSPxNvkqhuI28bri4zuF7xQFV2QuDXBUh1l_1KKP4gnf_1eHPyCRaZ1CCt3eBIAjzHRd44rbwJ8StAOwSfWVJ8cZn9Cv5BY2YvM-B0nWX_1h3EoWf0cOruX-ZTN-PnXRE5Dbfv8s88tjauwQNcPVnOwpDal56rdnqS4wkHm52FOrJYCg71pyhxhL_1pK5G2Q_1xmHP14jcqf0sqfHH-M9NPaC-ab1cMKAMVGWeWI5XI3zhWRZtGn1iX354oBmVMd2AKtgR4YrwSe2hVfB3opsut6oOMe5lgmL5HsSdzDp0bHOlKGdkhR-_1caWT3vN05ohcUF5ulolcyIjDJN6jPR-8qgHFMDNYrz2F3Xzz4xBmr8S3caTEik6EJxYysF37CvjLP0QjV3Oy9n6Wz3wfc1opQEXl0xUNwqCoOmADp29YNDGXUl34xIvnFcrn1LmaXH78tGSpFuqm_1k9WEB-4QNgKW1Xr_1rk-El6IN9FH7_1aZO1justpzsk2gf8s8wooFqFSPbbHLNDq0u99e2NEraHVyRv-c3nAcjwnu0L--1J9gIxpkZMHnNMWgCt3Z0q0FToYTQPGZLdSG8V_1Xhu_11GPVKyGd59uOG49f--ix0hWy2U_1NwCd3ITSj0r_19dtcJOaUcNrWKok21YuiS_1Uk_14wpSqxOaaXVS2qe2M_1B7LtSOxK-czdDyI0r-wHcmBMRyLjTPgexxNYg&btnG=Search%20by%20image">Google Reverse Image Search</a> na ten obrázok, kde "autor" stojí pri svojom novom aute, pozrite si to sami :)</p>
<p>Aby som to teda zhrnul. Nemám nič proti kasínam, to je každého osobná vec ako míňa svoje peniaze. Dokonca nemám ani nič proti autorovi tohoto článku, aj keď zneužíva to, že niektorí ľudia sú rozumovo slabší a neuvedomia si, o čo ide. Túto sériu článkov beriem s nadhľadom a ako cvičenie v štatistike a Pythone. Preto ak niekto nájde niekde nejakú chybu alebo nezrovnalosť, dajte mi vedieť.</p>
Python Pip Crashes2014-02-21T00:00:00+00:002014-02-21T00:00:00+00:00https://mirosval.sk/blog/2014/pip-crashes/<p>I was upgrading my disk to SSD the other day, and to make sure all my data fit, I've done some
cleaning up, among others I've run <code>brew cleanup</code> which deletes old formulae. But apparently
somewhere along the way that has broken my Python install. Trying to run Flask server would produce
a cryptic error:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>ImportError: No module named itsdangerous
</span></code></pre>
<p>Hmm okay, lets try to install it:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> % pip3 install itsdangerous
</span><span> Downloading/unpacking itsdangerous
</span><span> Downloading itsdangerous-0.23.tar.gz (46kB): 46kB downloaded
</span><span> [1] 48739 segmentation fault pip3 install itsdangerous
</span></code></pre>
<p>Mmmkaay, what? I've tried to reinstall python to no avail, the search did not show anything either...
The segfault looked like this:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Process: Python [61249]
</span><span>Path: /usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/Resources/Python.app/Contents/MacOS/Python
</span><span>Identifier: Python
</span><span>Version: 3.3.4 (3.3.4)
</span><span>Code Type: X86-64 (Native)
</span><span>Parent Process: sudo [61248]
</span><span>Responsible: Terminal [416]
</span><span>User ID: 0
</span><span>
</span><span>Date/Time: 2014-02-21 12:35:15.725 +0100
</span><span>OS Version: Mac OS X 10.9.1 (13B42)
</span><span>Report Version: 11
</span><span>
</span><span>Crashed Thread: 0 Dispatch queue: com.apple.main-thread
</span><span>
</span><span>Exception Type: EXC_BAD_ACCESS (SIGSEGV)
</span><span>Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000011
</span><span>
</span><span>VM Regions Near 0x11:
</span><span>-->
</span><span> __TEXT 00000001084d7000-00000001084d9000 [ 8K] r-x/rwx SM=COW /usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/Resources/Python.app/Contents/MacOS/Python
</span><span>
</span><span>Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
</span><span>0 libcrypto.1.0.0.dylib 0x0000000108f6d1ac EVP_PKEY_CTX_dup + 28
</span><span>1 libcrypto.1.0.0.dylib 0x0000000108f61565 EVP_MD_CTX_copy_ex + 325
</span><span>2 _hashlib.so 0x0000000108e88933 locked_EVP_MD_CTX_copy + 78
</span><span>3 _hashlib.so 0x0000000108e887f0 EVP_hexdigest + 51
</span><span>4 org.python.python 0x0000000108570bd2 PyEval_EvalFrameEx + 16052
</span><span>5 org.python.python 0x000000010857512d fast_function + 186
</span><span>6 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>7 org.python.python 0x000000010857512d fast_function + 186
</span><span>8 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>9 org.python.python 0x000000010857512d fast_function + 186
</span><span>10 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>11 org.python.python 0x000000010857512d fast_function + 186
</span><span>12 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>13 org.python.python 0x000000010857512d fast_function + 186
</span><span>14 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>15 org.python.python 0x000000010857512d fast_function + 186
</span><span>16 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>17 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>18 org.python.python 0x00000001085751a2 fast_function + 303
</span><span>19 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>20 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>21 org.python.python 0x00000001085751a2 fast_function + 303
</span><span>22 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>23 org.python.python 0x000000010857512d fast_function + 186
</span><span>24 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>25 org.python.python 0x000000010857512d fast_function + 186
</span><span>26 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>27 org.python.python 0x000000010857512d fast_function + 186
</span><span>28 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>29 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>30 org.python.python 0x0000000108506099 function_call + 345
</span><span>31 org.python.python 0x00000001084ea67a PyObject_Call + 111
</span><span>32 org.python.python 0x0000000108571486 PyEval_EvalFrameEx + 18280
</span><span>33 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>34 org.python.python 0x00000001085751a2 fast_function + 303
</span><span>35 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>36 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>37 org.python.python 0x00000001085751a2 fast_function + 303
</span><span>38 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>39 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>40 org.python.python 0x00000001085751a2 fast_function + 303
</span><span>41 org.python.python 0x0000000108570c92 PyEval_EvalFrameEx + 16244
</span><span>42 org.python.python 0x000000010856cb93 PyEval_EvalCodeEx + 1579
</span><span>43 org.python.python 0x000000010856c562 PyEval_EvalCode + 63
</span><span>44 org.python.python 0x000000010858d6cf run_mod + 58
</span><span>45 org.python.python 0x000000010858d8e4 PyRun_FileExFlags + 142
</span><span>46 org.python.python 0x000000010858d1b9 PyRun_SimpleFileExFlags + 875
</span><span>47 org.python.python 0x000000010859eaab Py_Main + 3123
</span><span>48 org.python.python 0x00000001084d8e3f 0x1084d7000 + 7743
</span><span>49 libdyld.dylib 0x00007fff91ea55fd start + 1
</span><span>
</span><span>Thread 1:
</span><span>0 libsystem_kernel.dylib 0x00007fff98b49e6a __workq_kernreturn + 10
</span><span>1 libsystem_pthread.dylib 0x00007fff8d74af08 _pthread_wqthread + 330
</span><span>2 libsystem_pthread.dylib 0x00007fff8d74dfb9 start_wqthread + 13
</span><span>
</span><span>Thread 2:: Dispatch queue: com.apple.libdispatch-manager
</span><span>0 libsystem_kernel.dylib 0x00007fff98b4a662 kevent64 + 10
</span><span>1 libdispatch.dylib 0x00007fff9363143d _dispatch_mgr_invoke + 239
</span><span>2 libdispatch.dylib 0x00007fff93631152 _dispatch_mgr_thread + 52
</span><span>
</span><span>Thread 3:
</span><span>0 libsystem_kernel.dylib 0x00007fff98b49e6a __workq_kernreturn + 10
</span><span>1 libsystem_pthread.dylib 0x00007fff8d74af08 _pthread_wqthread + 330
</span><span>2 libsystem_pthread.dylib 0x00007fff8d74dfb9 start_wqthread + 13
</span><span>
</span><span>Thread 0 crashed with X86 Thread State (64-bit):
</span><span> rax: 0x0000000000000001 rbx: 0x00007fff57726b00 rcx: 0x0000000000000010 rdx: 0xffffffffffffffd4
</span><span> rdi: 0x00007fe6ca7bde20 rsi: 0x00007fe6ca60d490 rbp: 0x00007fff57726a90 rsp: 0x00007fff57726a70
</span><span> r8: 0x0000000000000006 r9: 0x00007fe6ca400000 r10: 0x0000000069793538 r11: 0xffffffffffe9b3a0
</span><span> r12: 0x00007fff7c009400 r13: 0x00000001092c0f38 r14: 0x00007fe6ca7bde20 r15: 0x0000000000000000
</span><span> rip: 0x0000000108f6d1ac rfl: 0x0000000000010202 cr2: 0x0000000000000011
</span><span>
</span></code></pre>
<p>So libcrypto.dylib is at fault here... Lets see, Homebrew has an option to use Python
with brewed OpenSSL instead:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> % brew info python3
</span><span>
</span><span>python3: stable 3.3.4, HEAD
</span><span>http://www.python.org/
</span><span>Not installed
</span><span>From: https://github.com/Homebrew/homebrew/commits/master/Library/Formula/python3.rb
</span><span>==> Dependencies
</span><span>Build: pkg-config ✔
</span><span>Recommended: readline ✔, sqlite ✔, gdbm ✔, xz ✔
</span><span>==> Options
</span><span>--quicktest
</span><span> Run `make quicktest` after the build
</span><span>--universal
</span><span> Build a universal binary
</span><span>--with-brewed-openssl
</span><span> Use Homebrew's openSSL instead of the one from OS X
</span><span>--with-brewed-tk
</span><span> Use Homebrew's Tk (has optional Cocoa and threads support)
</span><span>--without-gdbm
</span><span> Build without gdbm support
</span><span>--without-readline
</span><span> Build without readline support
</span><span>--without-sqlite
</span><span> Build without sqlite support
</span><span>--without-xz
</span><span> Build without xz support
</span><span>--HEAD
</span><span> install HEAD version
</span><span>==> Caveats
</span><span>Setuptools and Pip have been installed. To update them
</span><span> pip3 install --upgrade setuptools
</span><span> pip3 install --upgrade pip
</span><span>
</span><span>You can install Python packages with
</span><span> `pip3 install <your_favorite_package>`
</span><span>
</span><span>They will install into the site-package directory
</span><span> /usr/local/lib/python3.3/site-packages
</span><span>
</span><span>See: https://github.com/Homebrew/homebrew/wiki/Homebrew-and-Python
</span><span>
</span></code></pre>
<p>So I ran:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> % brew uninstall python3
</span><span> % brew install python3 --with-brewed-openssl
</span></code></pre>
<p>And that seemed to do the trick for the pip segfaults, however it did not solve my original
problem. To do that I've uninstalled all of the Flask and related packages using pip3 and then
re-installed them in the virtual environment instead.</p>
Macbook Pro SSD Upgrade2014-02-20T00:00:00+00:002014-02-20T00:00:00+00:00https://mirosval.sk/blog/2014/ssd-upgrade/<p>I have a Macbook Pro from mid 2010. Its coming up on 4 years, so some would say that it's quite old,
but I've upgraded its RAM to 8GB and it has been working for me quite nicely until recently when I felt
that it's starting to get slow.</p>
<h3 id="dust-cleaning">Dust cleaning</h3>
<p>The first problem appeared in the summer, when I've installed Mavericks. At the time I've been using
Google Chrome as my primary web browser and it has become unbearable. The machine would run hot and
with fans at full speed with a lot of noise. So at the time I switched back to Safari which helped
tremendously, but Safari has many other shortcomings (Horrible PDF support, Reddit's RES didn't work, etc.)
so I wanted to switch back to Chrome, and now after Christmas they seem to have released a new version
(I'm using the Beta channel, so now it's 33) which does not suffer from the performance issues of the
old one, so I switched back and everything seemed fine, with just occasional fan rampage.</p>
<p>But it still bothered me, it seemed that the fans would go crazy for no reason (low CPU and energy use,
and still, fans were running at 6000rpm). So I've removed the back cover and cleaned all the dust from
the fans, that has helped incredibly, now it runs much cooler (up to 10°C cooler) and much more
quiet (most of the time just 2000rpm).</p>
<h3 id="new-ssd">New SSD</h3>
<p>I still wanted to improve the performance, and the SSD looked like a good option. After looking it
up briefly I decided on Samsung 840 EVO 250GB SSD, it was faster than what my Macbook could handle
(SATA III vs SATA II on my Macbook Pro), but not much more expensive than the lower options. At about
150€ it wasn't too expensive either. I got the Laptop Upgrade kit.</p>
<p>The kit comes with just a SATA to USB connector and some useless CD with software. It does not have
any screw drivers, but I had that covered. You only need a Phillips screwdriver luckily.</p>
<p>The annoying part was that my HD was 500GB and I needed to reduce that to about 200GB if I wanted
to fit into the SSD comfortably. I've moved all my photos to an external HDD, that took some good
3-4 hours.</p>
<p>I first formatted the SSD to <code>Mac OS Extended Journaled</code> and to move the filesystem to the SSD
I've used the <a href="http://www.bombich.com/">Carbon Copy Cloner</a>. It allows you to create the Mavericks
recovery partition on the drive as well, so you could just use that to make a clean install. But
I've copied my whole system over using the CCC, about 200GB took around 4 hours to complete.</p>
<p>I've encountered an issue with Time Machine local backups, they will use up the empty space on
the drive, so I've deleted almost 300GB of data and only had 70GB free on a 500GB drive. But it
was sufficient to just run the Time Machine and it would remove these local backups during the
Preparing Backup phase.</p>
<p>The actual HW swap went well, the SSD is much lighter and a bit smaller than the original HDD.
I've used the <a href="http://www.ifixit.com/Guide/MacBook+Pro+15-Inch+Unibody+Mid+2010+Hard+Drive+Replacement/3030">iFixit guide</a>,
but I haven't removed the battery and to unscrew the HDD screws on the side I've just used pliers
to unscrew them carefully and then screw them back on the SSD. This is necessary to hold it in
place, since it is slightly smaller than the original.</p>
<p>After the replacement I've used <a href="http://www.cindori.org/software/trimenabler/">Trim Enabler</a> to
enable TRIM support, because apparently Apple only enables TRIM on their own SSDs by default.</p>
<h3 id="performance">Performance</h3>
<p>I have measured my disk performance before and after the upgrade using <a href="https://itunes.apple.com/us/app/blackmagic-disk-speed-test/id425264550?mt=12">Blackmagic Disk Speed Test</a>
and this is what I got:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Before:
</span><span> Write: 42.5 MB/s Read: 39.9 MB/s
</span><span>After:
</span><span> Write: 202.0 MB/s Read: 256.5 MB/s
</span></code></pre>
<p>Subjectively it feels a lot faster, especially opening apps is almost instantaneous. The
main bottleneck now is probably the 3Gb/s SATA II interface, because the SSD should give
around 500MB/s on SATA III. Over all I think this was a good investment and a relatively
painless upgrade (apart from all the slow copying, ugh).</p>
<p>Here are the Speed Test Screen Shots:</p>
<h4 id="before-upgrade">Before upgrade</h4>
<p><img src="https://mirosval.sk/blog/2014/ssd-upgrade/old.png" alt="Before SSD Upgrade" /></p>
<h4 id="after-upgrade">After upgrade</h4>
<p><img src="https://mirosval.sk/blog/2014/ssd-upgrade/new.png" alt="After SSD Upgrade" /></p>
Performance of the unwrapIris()2014-02-11T00:00:00+00:002014-02-11T00:00:00+00:00https://mirosval.sk/blog/2014/unwrap-performance/<p>After the performance improvement from yesterday, I wanted to try some more things,
because the speed of this was still not satisfactory (I spent an hour processing 2000 images).</p>
<p>So I've whipped up line_profiler again:</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#bf616a;">kernprof.py -l</span><span> main.py && </span><span style="color:#bf616a;">python -m</span><span> line_profiler main.py.lprof
</span></code></pre>
<p>This gave me the following trace:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>File: main.py
</span><span>Function: main at line 10
</span><span>Total time: 23.1351 s
</span><span>
</span><span>Line # Hits Time Per Hit % Time Line Contents
</span><span>==============================================================
</span><span> 10 @profile
</span><span> 11 def main():
</span><span> 12 1 167798 167798.0 0.7 cv2.namedWindow("Eye")
</span><span> 13 1 28931 28931.0 0.1 cv2.namedWindow("Iris")
</span><span> 14
</span><span> 15 1 21107 21107.0 0.1 cv2.moveWindow("Eye", 0, 0)
</span><span> 16 1 27887 27887.0 0.1 cv2.moveWindow("Iris", 800, 0)
</span><span> 17
</span><span> 18 1 6 6.0 0.0 i = 0
</span><span> 19 12 247887 20657.2 1.1 for f in os.listdir(data_dir):
</span><span> 20 12 240693 20057.8 1.0 image = cv2.imread(os.path.join(data_dir, f))
</span><span> 21 12 106 8.8 0.0 if image is None:
</span><span> 22 1 3 3.0 0.0 continue
</span><span> 23
</span><span> 24 11 31933 2903.0 0.1 gray = cv2.cvtColor(image, cv2.cv.CV_BGR2GRAY)
</span><span> 25
</span><span> 26 11 179806 16346.0 0.8 pupil = functions.findPupil(gray, show=True)
</span><span> 27
</span><span> 28 11 51 4.6 0.0 if not pupil:
</span><span> 29 print("No pupil found")
</span><span> 30 continue
</span><span> 31
</span><span> 32 11 39 3.5 0.0 center, ellipse = pupil
</span><span> 33
</span><span> 34 11 33 3.0 0.0 pupil_width = ellipse[1][0]
</span><span> 35
</span><span> 36 11 1755948 159631.6 7.6 iris_radius = functions.findIris(gray, center, pupil_width, pupil_ellipse=ellipse, show=True)
</span><span> 37 11 77 7.0 0.0 if iris_radius > 0:
</span><span> 38 11 20144018 1831274.4 87.1 polar = functions.unwrapIris(gray, center, iris_radius, show=True)
</span><span> 39
</span><span> 40 11 51121 4647.4 0.2 cv2.imwrite(os.path.join(dest_dir, f.replace('.png', '') + "_polar.png"), polar)
</span><span> 41 11 369 33.5 0.0 print("Written %s" % f)
</span><span> 42
</span><span> 43 11 237059 21550.8 1.0 if cv2.waitKey(1) == ord('q'):
</span><span> 44 break
</span><span> 45
</span><span> 46 11 108 9.8 0.0 if i == 10:
</span><span> 47 1 61 61.0 0.0 break
</span><span> 48
</span><span> 49 10 37 3.7 0.0 i += 1
</span></code></pre>
<p>Hmmm, 87% of the time is spent in <code>unwrapIris()</code>. Lets take a look at what that looks like:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">unwrapIris</span><span>(</span><span style="color:#bf616a;">image</span><span>, </span><span style="color:#bf616a;">iris_center</span><span>, </span><span style="color:#bf616a;">iris_radius</span><span>, </span><span style="color:#bf616a;">nsamples</span><span>=</span><span style="color:#d08770;">360</span><span>, </span><span style="color:#bf616a;">show</span><span>=</span><span style="color:#d08770;">False</span><span>):
</span><span> samples = np.</span><span style="color:#bf616a;">linspace</span><span>(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">2 </span><span>* np.pi, nsamples)[:-</span><span style="color:#d08770;">1</span><span>]
</span><span> polar = np.</span><span style="color:#bf616a;">zeros</span><span>((iris_radius, nsamples), </span><span style="color:#bf616a;">dtype</span><span>=np.uint8)
</span><span>
</span><span> </span><span style="color:#b48ead;">for </span><span>r </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(iris_radius):
</span><span> </span><span style="color:#b48ead;">for </span><span>theta </span><span style="color:#b48ead;">in </span><span>samples:
</span><span> x = r * np.</span><span style="color:#bf616a;">cos</span><span>(theta) + iris_center[</span><span style="color:#d08770;">0</span><span>]
</span><span> y = r * np.</span><span style="color:#bf616a;">sin</span><span>(theta) + iris_center[</span><span style="color:#d08770;">1</span><span>]
</span><span> </span><span style="color:#b48ead;">try</span><span>:
</span><span> polar[r][theta * nsamples / </span><span style="color:#d08770;">2.0 </span><span>/ np.pi] = image[y][x]
</span><span> </span><span style="color:#b48ead;">except </span><span>IndexError:
</span><span> polar[r][theta * nsamples / </span><span style="color:#d08770;">2.0 </span><span>/ np.pi] = </span><span style="color:#d08770;">0
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>show:
</span><span> cv2.</span><span style="color:#bf616a;">imshow</span><span>("</span><span style="color:#a3be8c;">Iris</span><span>", polar)
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>polar
</span></code></pre>
<p>And its profile:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>File: functions.py
</span><span>Function: unwrapIris at line 145
</span><span>Total time: 34.5483 s
</span><span>
</span><span>Line # Hits Time Per Hit % Time Line Contents
</span><span>==============================================================
</span><span> 145 @profile
</span><span> 146 def unwrapIris(image, iris_center, iris_radius, nsamples=360, show=False):
</span><span> 147 11 1305 118.6 0.0 samples = np.linspace(0, 2 * np.pi, nsamples)[:-1]
</span><span> 148 11 294 26.7 0.0 polar = np.zeros((iris_radius, nsamples), dtype=np.uint8)
</span><span> 149
</span><span> 150 1331 4389 3.3 0.0 for r in range(iris_radius):
</span><span> 151 475200 1935876 4.1 5.6 for theta in samples:
</span><span> 152 473880 10479969 22.1 30.3 x = r * np.cos(theta) + iris_center[0]
</span><span> 153 473880 10187509 21.5 29.5 y = r * np.sin(theta) + iris_center[1]
</span><span> 154 473880 1476839 3.1 4.3 try:
</span><span> 155 473880 10431987 22.0 30.2 polar[r][theta * nsamples / 2.0 / np.pi] = image[y][x]
</span><span> 156 except IndexError:
</span><span> 157 polar[r][theta * nsamples / 2.0 / np.pi] = 0
</span><span> 158
</span><span> 159 11 28 2.5 0.0 if show:
</span><span> 160 11 30015 2728.6 0.1 cv2.imshow("Iris", polar)
</span><span> 161
</span><span> 162 11 52 4.7 0.0 return polar
</span></code></pre>
<p>So we can see that the load here is distributed between 3 lines, 2 trigonometric calculations
and one array access and some arithmetic. But the main problem is that these 3 lines are run for
each pixel, for 10 images processed in this sample this is 475 200 hits. If I have learned anything
about Python performance in the past year, it is to move all loops to an external library if possible.
In this case we'll look at how could this be moved over to either Numpy or OpenCV. Since these
libraries are written in C or C++, they can perform the same loop much faster than plain Python.</p>
<p>Thanks to Numpy's operator overloading, it is possible to write code that looks really weird at the
first glance, if you're used to loops from other languages. Look at the lines 6 and 7. <code>magnitude</code>
is an array, <code>angle</code> is an array and <code>iris_center</code> is an <code>int</code> as it is used here. Numpy can
calculate this correctly and the loops are now in the library, yay!</p>
<p>So what changed? Line 4 makes two arrays, one with angles and one with magnitudes, both of dimensions
<code>iris_radius&times;nsamples</code> which is about <code>359x130</code> for most of my images. The angle array is
basically just a single row of radian values from 0 to 2pi repeated 130 times, and magnitude is
a column from 0 to 129 repeated horizontally 359 times.</p>
<p>The lines 6 and 7 convert these two arrays to X,Y coordinates in the image, from which we will be
sampling later. <code>convertMaps()</code> converts the them further to improve performance of <code>cv2.remap()</code>.
And finally <code>cv2.remap()</code> maps these coordinate maps from the original image into the polar image.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">unwrapIris</span><span>(</span><span style="color:#bf616a;">image</span><span>, </span><span style="color:#bf616a;">iris_center</span><span>, </span><span style="color:#bf616a;">iris_radius</span><span>, </span><span style="color:#bf616a;">nsamples</span><span>=</span><span style="color:#d08770;">360</span><span>, </span><span style="color:#bf616a;">show</span><span>=</span><span style="color:#d08770;">False</span><span>):
</span><span> samples = np.</span><span style="color:#bf616a;">linspace</span><span>(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">2 </span><span>* np.pi, nsamples)[:-</span><span style="color:#d08770;">1</span><span>]
</span><span>
</span><span> angle, magnitude = np.</span><span style="color:#bf616a;">meshgrid</span><span>(samples, np.</span><span style="color:#bf616a;">arange</span><span>(iris_radius))
</span><span>
</span><span> x = magnitude * np.</span><span style="color:#bf616a;">cos</span><span>(angle) + iris_center[</span><span style="color:#d08770;">0</span><span>]
</span><span> y = magnitude * np.</span><span style="color:#bf616a;">sin</span><span>(angle) + iris_center[</span><span style="color:#d08770;">1</span><span>]
</span><span>
</span><span> x, y = cv2.</span><span style="color:#bf616a;">convertMaps</span><span>(x.</span><span style="color:#bf616a;">astype</span><span>('</span><span style="color:#a3be8c;">float32</span><span>'), y.</span><span style="color:#bf616a;">astype</span><span>('</span><span style="color:#a3be8c;">float32</span><span>'), cv2.</span><span style="color:#bf616a;">CV_32FC1</span><span>)
</span><span> </span><span style="color:#b48ead;">return </span><span>cv2.</span><span style="color:#bf616a;">remap</span><span>(image, x, y, cv2.</span><span style="color:#bf616a;">INTER_LINEAR</span><span>)
</span></code></pre>
<p>And now lets take a look at what that did to the performance:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>File: functions.py
</span><span>Function: unwrapIris at line 145
</span><span>Total time: 0.090496 s
</span><span>
</span><span>Line # Hits Time Per Hit % Time Line Contents
</span><span>==============================================================
</span><span> 145 @profile
</span><span> 146 def unwrapIris(image, iris_center, iris_radius, nsamples=360, show=False):
</span><span> 147 11 1445 131.4 1.6 samples = np.linspace(0, 2 * np.pi, nsamples)[:-1]
</span><span> 148
</span><span> 149 11 11660 1060.0 12.9 angle, magnitude = np.meshgrid(samples, np.arange(iris_radius))
</span><span> 150
</span><span> 151 11 30335 2757.7 33.5 x = magnitude * np.cos(angle) + iris_center[0]
</span><span> 152 11 29815 2710.5 32.9 y = magnitude * np.sin(angle) + iris_center[1]
</span><span> 153
</span><span> 154 11 5195 472.3 5.7 x, y = cv2.convertMaps(x.astype('float32'), y.astype('float32'), cv2.CV_32FC1)
</span><span> 155 11 12046 1095.1 13.3 return cv2.remap(image, x, y, cv2.INTER_LINEAR)
</span><span>
</span><span>File: main.py
</span><span>Function: main at line 10
</span><span>Total time: 3.99889 s
</span><span>
</span><span>Line # Hits Time Per Hit % Time Line Contents
</span><span>==============================================================
</span><span> 10 @profile
</span><span> 11 def main():
</span><span> 12 1 160876 160876.0 4.0 cv2.namedWindow("Eye")
</span><span> 13 1 11790 11790.0 0.3 cv2.namedWindow("Iris")
</span><span> 14
</span><span> 15 1 26153 26153.0 0.7 cv2.moveWindow("Eye", 0, 0)
</span><span> 16 1 25946 25946.0 0.6 cv2.moveWindow("Iris", 800, 0)
</span><span> 17
</span><span> 18 1 9 9.0 0.0 i = 0
</span><span> 19 12 3073 256.1 0.1 for f in os.listdir(data_dir):
</span><span> 20 12 88505 7375.4 2.2 image = cv2.imread(os.path.join(data_dir, f))
</span><span> 21 12 124 10.3 0.0 if image is None:
</span><span> 22 1 4 4.0 0.0 continue
</span><span> 23
</span><span> 24 11 9028 820.7 0.2 gray = cv2.cvtColor(image, cv2.cv.CV_BGR2GRAY)
</span><span> 25
</span><span> 26 11 38994 3544.9 1.0 pupil = functions.findPupil(gray, show=True)
</span><span> 27
</span><span> 28 11 187 17.0 0.0 if not pupil:
</span><span> 29 print("No pupil found")
</span><span> 30 continue
</span><span> 31
</span><span> 32 11 68 6.2 0.0 center, ellipse = pupil
</span><span> 33
</span><span> 34 11 63 5.7 0.0 pupil_width = ellipse[1][0]
</span><span> 35
</span><span> 36 11 3202228 291111.6 80.1 iris_radius = functions.findIris(gray, center, pupil_width, pupil_ellipse=ellipse, show=True)
</span><span> 37 11 100 9.1 0.0 if iris_radius > 0:
</span><span> 38 11 91645 8331.4 2.3 polar = functions.unwrapIris(gray, center, iris_radius, show=True)
</span><span> 39
</span><span> 40 11 28406 2582.4 0.7 cv2.imwrite(os.path.join(dest_dir, f.replace('.png', '') + "_polar.png"), polar)
</span><span> 41 11 563 51.2 0.0 print("Written %s" % f)
</span><span> 42
</span><span> 43 11 310859 28259.9 7.8 if cv2.waitKey(1) == ord('q'):
</span><span> 44 break
</span><span> 45
</span><span> 46 11 120 10.9 0.0 if i == 10:
</span><span> 47 1 92 92.0 0.0 break
</span><span> 48
</span><span> 49 10 57 5.7 0.0 i += 1
</span></code></pre>
<p>Nice! Now <code>unwrapIris()</code> is taking just 2.3% of the total time! And we have reduced time needed to process
10 images from ~34s to about 4s, that's an order of magnitude improvement! Now the <code>findIris()</code> is the
slowest, maybe next time we'll look at that.</p>
Performance of the getOrientationAndMagnitude()2014-02-10T00:00:00+00:002014-02-10T00:00:00+00:00https://mirosval.sk/blog/2014/orientation-performance/<p>I normally don't do premature performance optimizations, and I was planning on optimizing the whole eye tracker
later, when I felt I had all of the functionality I wanted, but from time to time, you still want to check your
code, particularly when it's suspiciously slow. So I did examine the code I had written so far using
<a href="http://packages.python.org/line_profiler/">line_profiler</a> (Abysmal documentation btw.) like so:</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#bf616a;">kernprof.py -l</span><span> test.py && </span><span style="color:#bf616a;">python -m</span><span> line_profiler test.py.lprof
</span></code></pre>
<p>I have found that the most time is being spent in one function, <code>getOrientationAndMagnitude</code>, you'll recall it
looked like this:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">getOrientationAndMagnitude</span><span>(</span><span style="color:#bf616a;">image</span><span>, </span><span style="color:#bf616a;">show</span><span>=</span><span style="color:#d08770;">False</span><span>):
</span><span> sobelHorizontal = cv2.</span><span style="color:#bf616a;">Sobel</span><span>(image, cv2.</span><span style="color:#bf616a;">CV_32F</span><span>, </span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">0</span><span>)
</span><span> sobelVertical = cv2.</span><span style="color:#bf616a;">Sobel</span><span>(image, cv2.</span><span style="color:#bf616a;">CV_32F</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">1</span><span>)
</span><span>
</span><span> h = sobelHorizontal
</span><span> v = sobelVertical
</span><span>
</span><span> orientation = np.</span><span style="color:#bf616a;">empty</span><span>(image.shape)
</span><span> magnitude = np.</span><span style="color:#bf616a;">empty</span><span>(image.shape)
</span><span>
</span><span> height, width = h.shape
</span><span> </span><span style="color:#b48ead;">for </span><span>y </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(height):
</span><span> </span><span style="color:#b48ead;">for </span><span>x </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(width):
</span><span> orientation[y][x] = cv2.</span><span style="color:#bf616a;">fastAtan2</span><span>(h[y][x], v[y][x])
</span><span>
</span><span> magnitude = cv2.</span><span style="color:#bf616a;">magnitude</span><span>(h, v)
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>orientation, magnitude
</span></code></pre>
<p>I wrote this code some time ago, when I was less familiar with OpenCV than I am now, and you can quickly see
the hotspot. Yes, the line <code>14</code>. I probably did it like this because I hadn't noticed the
<a href="http://docs.opencv.org/modules/core/doc/operations_on_arrays.html#phase"><code>phase()</code></a> function OpenCV has.</p>
<p>But this will serve the purpose of showing how such a small oversight can have a dramatic effect on performance.
Armed with the <a href="http://docs.opencv.org/modules/core/doc/operations_on_arrays.html#phase"><code>phase()</code></a> function,
we can do the following:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">getOrientationAndMagnitude</span><span>(</span><span style="color:#bf616a;">image</span><span>, </span><span style="color:#bf616a;">show</span><span>=</span><span style="color:#d08770;">False</span><span>):
</span><span> h = cv2.</span><span style="color:#bf616a;">Sobel</span><span>(image, cv2.</span><span style="color:#bf616a;">CV_32F</span><span>, </span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">0</span><span>)
</span><span> v = cv2.</span><span style="color:#bf616a;">Sobel</span><span>(image, cv2.</span><span style="color:#bf616a;">CV_32F</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">1</span><span>)
</span><span>
</span><span> orientation = cv2.</span><span style="color:#bf616a;">phase</span><span>(h, v, </span><span style="color:#bf616a;">angleInDegrees</span><span>=</span><span style="color:#d08770;">True</span><span>)
</span><span> magnitude = cv2.</span><span style="color:#bf616a;">magnitude</span><span>(h, v)
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>orientation, magnitude
</span></code></pre>
<p>Now this much shorter and simpler function will also run much faster, thanks to OpenCV. And here's how I tested
it:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>cProfile
</span><span>
</span><span>image = cv2.</span><span style="color:#bf616a;">imread</span><span>('</span><span style="color:#a3be8c;">eye.png</span><span>')
</span><span>image = cv2.</span><span style="color:#bf616a;">cvtColor</span><span>(image, cv2.cv.</span><span style="color:#bf616a;">CV_BGR2GRAY</span><span>)
</span><span>image2 = np.</span><span style="color:#bf616a;">copy</span><span>(image)
</span><span>
</span><span>cProfile.</span><span style="color:#bf616a;">runctx</span><span>("</span><span style="color:#a3be8c;">refGetOrientationAndMagnitude(image)</span><span>", </span><span style="color:#bf616a;">globals</span><span>=</span><span style="color:#96b5b4;">globals</span><span>(), </span><span style="color:#bf616a;">locals</span><span>=</span><span style="color:#96b5b4;">locals</span><span>())
</span><span>cProfile.</span><span style="color:#bf616a;">runctx</span><span>("</span><span style="color:#a3be8c;">newGetOrientationAndMagnitude(image)</span><span>", </span><span style="color:#bf616a;">globals</span><span>=</span><span style="color:#96b5b4;">globals</span><span>(), </span><span style="color:#bf616a;">locals</span><span>=</span><span style="color:#96b5b4;">locals</span><span>())
</span></code></pre>
<p>Which yielded the following results:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>
</span><span>201309 function calls in 1.022 seconds
</span><span>
</span><span>Ordered by: standard name
</span><span>
</span><span>ncalls tottime percall cumtime percall filename:lineno(function)
</span><span> 1 0.000 0.000 1.022 1.022 <string>:1(<module>)
</span><span> 1 0.823 0.823 1.022 1.022 test.py:7(refGetOrientationAndMagnitude)
</span><span> 2 0.004 0.002 0.004 0.002 {cv2.Sobel}
</span><span>201000 0.192 0.000 0.192 0.000 {cv2.fastAtan2}
</span><span> 1 0.001 0.001 0.001 0.001 {cv2.magnitude}
</span><span> 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
</span><span> 2 0.000 0.000 0.000 0.000 {numpy.core.multiarray.empty}
</span><span> 301 0.002 0.000 0.002 0.000 {range}
</span><span>
</span><span>
</span><span>7 function calls in 0.004 seconds
</span><span>
</span><span>Ordered by: standard name
</span><span>
</span><span>ncalls tottime percall cumtime percall filename:lineno(function)
</span><span> 1 0.000 0.000 0.004 0.004 <string>:1(<module>)
</span><span> 1 0.000 0.000 0.003 0.003 test.py:26(candidateGetOrientationAndMagnitude)
</span><span> 2 0.002 0.001 0.002 0.001 {cv2.Sobel}
</span><span> 1 0.000 0.000 0.000 0.000 {cv2.magnitude}
</span><span> 1 0.001 0.001 0.001 0.001 {cv2.phase}
</span><span> 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
</span><span>
</span></code></pre>
<p>So 1.022s to 0.004, 255× faster with just replacing a double <code>for</code> loop with an OpenCV call.</p>
Unwrapping the iris2014-02-01T00:00:00+00:002014-02-01T00:00:00+00:00https://mirosval.sk/blog/2014/iris-unwrap/<p>In the previous installments we've looked at <a href="https://mirosval.sk/blog/2014/pupil-detection/">Pupil Detection</a>
and <a href="https://mirosval.sk/blog/2014/iris-detection/">Iris Detection</a>. Now we'll look at unwrapping the image
of the iris from a circular pattern to a rectangular one. We will use this later for some other
algorithms. </p>
<p>So by applying the above mentioned algorithms we got two bits of information: the center of the pupil and
the estimated radius of the iris. We can see them on the image below.</p>
<p><img src="https://mirosval.sk/blog/2014/iris-unwrap/001-original.png" alt="Pupil center and iris radius" /></p>
<p>In order to transform this circular iris into a rectangular image, we do the following:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>cv2
</span><span style="color:#b48ead;">import </span><span>numpy </span><span style="color:#b48ead;">as </span><span>np
</span><span style="color:#b48ead;">import </span><span>matplotlib.pyplot </span><span style="color:#b48ead;">as </span><span>plt
</span><span>
</span><span>image = cv2.</span><span style="color:#bf616a;">imread</span><span>('</span><span style="color:#a3be8c;">003/eye.png</span><span>')
</span><span>
</span><span>center = (</span><span style="color:#d08770;">376</span><span>, </span><span style="color:#d08770;">184</span><span>)
</span><span>iris_radius = </span><span style="color:#d08770;">115
</span><span>
</span><span>nsamples = </span><span style="color:#d08770;">360
</span><span>samples = np.</span><span style="color:#bf616a;">linspace</span><span>(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">2 </span><span>* np.pi, nsamples)[:-</span><span style="color:#d08770;">1</span><span>]
</span><span>
</span><span>polar = np.</span><span style="color:#bf616a;">zeros</span><span>((iris_radius, nsamples))
</span><span>
</span><span style="color:#b48ead;">for </span><span>r </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(iris_radius):
</span><span> </span><span style="color:#b48ead;">for </span><span>theta </span><span style="color:#b48ead;">in </span><span>samples:
</span><span> x = r * np.</span><span style="color:#bf616a;">cos</span><span>(theta) + center[</span><span style="color:#d08770;">0</span><span>]
</span><span> y = r * np.</span><span style="color:#bf616a;">sin</span><span>(theta) + center[</span><span style="color:#d08770;">1</span><span>]
</span><span>
</span><span> polar[r][theta * nsamples / </span><span style="color:#d08770;">2.0 </span><span>/ np.pi] = image[y][x][</span><span style="color:#d08770;">0</span><span>]
</span><span>
</span><span>plt.</span><span style="color:#bf616a;">figure</span><span>(</span><span style="color:#bf616a;">figsize</span><span>=(</span><span style="color:#d08770;">10</span><span>, </span><span style="color:#d08770;">5</span><span>))
</span><span>plt.</span><span style="color:#bf616a;">imshow</span><span>(polar, </span><span style="color:#bf616a;">cmap</span><span>='</span><span style="color:#a3be8c;">gray</span><span>')
</span></code></pre>
<p>We create an empty image with resolution <code>iris_radius * 360</code> and then we iterate through this image
and calculate the corresponding position in the circular image from the angle and the radius and copy
this value to the new image.</p>
<p><img src="https://mirosval.sk/blog/2014/iris-unwrap/002-unwrapped.png" alt="Pupil center and iris radius" /></p>
<p>This post is also available as an <a href="http://nbviewer.ipython.org/gist/mirosval/8754366">iPython Notebook</a></p>
Units Safari Extension2014-02-01T00:00:00+00:002014-02-01T00:00:00+00:00https://mirosval.sk/blog/2014/safari-units/<p>Did you know that there are still people in this world who use imperial units?
For some reason they refuse to switch and continue using this stupid system.
I requent Reddit and it's quite common to find these people there, so instead
of just bitching I decided to do something about it.</p>
<p>I have made this Safari Extension <a href="https://github.com/mirosval/Units-Safari-Extension">Units</a>,
<a href="https://github.com/mirosval/Units-Safari-Extension/blob/master/units.safariextz?raw=true">Download</a></p>
<p>It takes common expressions as <code>He is 5'4" tall</code> and translates them to <code>He is 5'4" (162.56 cm) tall</code>,
so now you know that <em>he</em> is quite short.</p>
<p>It can also convert some other units, such as miles to km, mpg to km/l and lbs to kg. Of course it
does have limitations, for example if somebody writes <code>He is 6 feet tall</code> or <code>He is 200</code> it does not
get translated, maybe I'll add it later, or if somebody wants to contribute, you're more than
welcome.</p>
<p>I've switched from Safari back to Chrome recently, so might be I port it there as well,
when I get sufficiently annoyed again.</p>
Basic Iris Detection2014-01-30T00:00:00+00:002014-01-30T00:00:00+00:00https://mirosval.sk/blog/2014/iris-detection/<p>Iris detection is significantly more difficult than pupil. Mostly because it is not so well defined, pupil is just a black disk essentially as far as a grayscale image is concerned, but the outer edge of the iris is not nearly as sharp as the outer edge of the pupil is. Also, there is a much higher probability of glints and reflections in the iris, since it is larger than the pupil and this can complicate the detection further. And on top of all that the upper and lower eyelids often cover up portions of the iris, which means that detecting the iris as a circle is going to be either difficult or impossible.</p>
<p>Despite all of these obstacles we have to do it somehow, so for this work, we are going to use the following:</p>
<ul>
<li>Start off with pupil location</li>
<li>Smooth down the image using median filter</li>
<li>Calculate Gradient Image</li>
<li>Sample lines from the pupil outside</li>
<li>Vote for the radius with greatest color change</li>
<li>Draw the iris estimate</li>
</ul>
<p>We begin with the location of the pupil as determined by the <a href="https://mirosval.sk/blog/2014/pupil-detection/">Pupil Detection</a>. The iris detection technique presented here is relying on the corectness of the pupil detection. It therefore very important for the pupil detection to be correct.</p>
<p><img src="https://mirosval.sk/blog/2014/iris-detection/001.png" alt="Pupil detection" /></p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">getOrientationAndMagnitude</span><span>(</span><span style="color:#bf616a;">image</span><span>, </span><span style="color:#bf616a;">show</span><span>=</span><span style="color:#d08770;">False</span><span>):
</span><span> sobelHorizontal = cv2.</span><span style="color:#bf616a;">Sobel</span><span>(image, cv2.</span><span style="color:#bf616a;">CV_32F</span><span>, </span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">0</span><span>)
</span><span> sobelVertical = cv2.</span><span style="color:#bf616a;">Sobel</span><span>(image, cv2.</span><span style="color:#bf616a;">CV_32F</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">1</span><span>)
</span><span>
</span><span> h = sobelHorizontal
</span><span> v = sobelVertical
</span><span>
</span><span> orientation = np.</span><span style="color:#bf616a;">empty</span><span>(image.shape)
</span><span> magnitude = np.</span><span style="color:#bf616a;">empty</span><span>(image.shape)
</span><span>
</span><span> height, width = h.shape
</span><span> </span><span style="color:#b48ead;">for </span><span>y </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(height):
</span><span> </span><span style="color:#b48ead;">for </span><span>x </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(width):
</span><span> orientation[y][x] = cv2.</span><span style="color:#bf616a;">fastAtan2</span><span>(h[y][x], v[y][x])
</span><span>
</span><span> magnitude = cv2.</span><span style="color:#bf616a;">magnitude</span><span>(h, v)
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>orientation, magnitude
</span></code></pre>
<p><img src="https://mirosval.sk/blog/2014/iris-detection/003-orientation.png" alt="Orientation" />
<img src="https://mirosval.sk/blog/2014/iris-detection/002-magnitude.png" alt="Magnitude" /></p>
<p>We know that the iris is going to be somewhere aroudn the pupil, so to get its radius, we are going to first get a couple of samples (lines that go from the center of the pupil) and then we are going to find the place where the change in intensity is the right orientation and magnitude. To be able to do that, we first need to calculate the orientation and magnitude fields of the image. That is what you can see above.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">getCircleSamples</span><span>(</span><span style="color:#bf616a;">center</span><span>=(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>), </span><span style="color:#bf616a;">radius</span><span>=</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#bf616a;">nPoints</span><span>=</span><span style="color:#d08770;">30</span><span>):
</span><span> s = np.</span><span style="color:#bf616a;">linspace</span><span>(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">2 </span><span>* np.pi, nPoints - </span><span style="color:#d08770;">1</span><span>)[:-</span><span style="color:#d08770;">1</span><span>]
</span><span> </span><span style="color:#65737e;"># print(np.sin(s))
</span><span>
</span><span> s = </span><span style="color:#96b5b4;">filter</span><span>(</span><span style="color:#b48ead;">lambda </span><span style="color:#bf616a;">x</span><span>: np.</span><span style="color:#bf616a;">abs</span><span>(np.</span><span style="color:#bf616a;">sin</span><span>(x)) < </span><span style="color:#d08770;">0.7</span><span>, s)
</span><span>
</span><span> </span><span style="color:#65737e;"># points
</span><span> P = [( radius * np.</span><span style="color:#bf616a;">cos</span><span>(t) + center[</span><span style="color:#d08770;">0</span><span>],
</span><span> radius * np.</span><span style="color:#bf616a;">sin</span><span>(t) + center[</span><span style="color:#d08770;">1</span><span>],
</span><span> np.</span><span style="color:#bf616a;">cos</span><span>(t),
</span><span> np.</span><span style="color:#bf616a;">sin</span><span>(t)) </span><span style="color:#b48ead;">for </span><span>t </span><span style="color:#b48ead;">in </span><span>s ]
</span><span> </span><span style="color:#b48ead;">return </span><span>P
</span></code></pre>
<p>This function returns a list of tuples, each representing 2 points that delimit a line going from the center of the circle outwards in regular intervals.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">getLineCoordinates</span><span>(</span><span style="color:#bf616a;">p1</span><span>, </span><span style="color:#bf616a;">p2</span><span>):
</span><span> (x1, y1) = p1
</span><span> (x2, y2) = p2
</span><span>
</span><span> points = []
</span><span> issteep = </span><span style="color:#96b5b4;">abs</span><span>(y2 - y1) > </span><span style="color:#96b5b4;">abs</span><span>(x2 - x1)
</span><span> </span><span style="color:#b48ead;">if </span><span>issteep:
</span><span> x1, y1 = y1, x1
</span><span> x2, y2 = y2, x2
</span><span> rev = </span><span style="color:#d08770;">False
</span><span> </span><span style="color:#b48ead;">if </span><span>x1 > x2:
</span><span> x1, x2 = x2, x1
</span><span> y1, y2 = y2, y1
</span><span> rev = </span><span style="color:#d08770;">True
</span><span> deltax = x2 - x1
</span><span> deltay = </span><span style="color:#96b5b4;">abs</span><span>(y2 - y1)
</span><span> error = </span><span style="color:#bf616a;">int</span><span>(deltax / </span><span style="color:#d08770;">2</span><span>)
</span><span> y = y1
</span><span> ystep = </span><span style="color:#d08770;">None
</span><span> </span><span style="color:#b48ead;">if </span><span>y1 < y2:
</span><span> ystep = </span><span style="color:#d08770;">1
</span><span> </span><span style="color:#b48ead;">else</span><span>:
</span><span> ystep = -</span><span style="color:#d08770;">1
</span><span> </span><span style="color:#b48ead;">for </span><span>x </span><span style="color:#b48ead;">in </span><span style="color:#96b5b4;">range</span><span>(x1, x2 + </span><span style="color:#d08770;">1</span><span>):
</span><span> </span><span style="color:#b48ead;">if </span><span>issteep:
</span><span> points.</span><span style="color:#bf616a;">append</span><span>([y, x])
</span><span> </span><span style="color:#b48ead;">else</span><span>:
</span><span> points.</span><span style="color:#bf616a;">append</span><span>([x, y])
</span><span> error -= deltay
</span><span> </span><span style="color:#b48ead;">if </span><span>error < </span><span style="color:#d08770;">0</span><span>:
</span><span> y += ystep
</span><span> error += deltax
</span><span> </span><span style="color:#65737e;"># Reverse the list if the coordinates were reversed
</span><span> </span><span style="color:#b48ead;">if </span><span>rev:
</span><span> points.</span><span style="color:#bf616a;">reverse</span><span>()
</span><span>
</span><span> retPoints = np.</span><span style="color:#bf616a;">array</span><span>(points)
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>retPoints
</span></code></pre>
<p>This function converts the coordinates of two points into a list of points that lay on the line between the two points. We will evaluate the gradient along this line, and for that we need a list of positions where to look.</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>The final algorithm has been removed at the request of the ITU.
</span></code></pre>
<p>Now to put it all together, we start with orientation and magnitude maps. Then sample the original image to get lines from the pupil out. We loop through these samples and find places where the gradient magnitude and orientation is just right and then add a vote for the radius at that location. In the end we just grab the radius with the most votes and draw an ellipse around it. In the image below, the purple circle is the estimated iris radius, the aqua circles are votes and the green lines are samples. You can see that the lines don't cover the whole circle. That is because it is probable that eyelids will cover top and bottom portion of the iris and therefore interfere with the algorithm, so we only consider the horizontal samples, as defined by <code>sin(x) < 0.7</code>.</p>
<p><img src="https://mirosval.sk/blog/2014/iris-detection/004-final.png" alt="Final image iris radius estimation" /></p>
<p>This is it for the iris detection, next time we'll look at unwrapping the iris to form a rectangular image.</p>
<p>This post is also available as an <a href="http://nbviewer.ipython.org/gist/mirosval/8752437">iPython Notebook</a></p>
Basic Pupil Detection2014-01-29T00:00:00+00:002014-01-29T00:00:00+00:00https://mirosval.sk/blog/2014/pupil-detection/<p>Pupil Detection will be the basis of my eye tracker. In the end it will rely on accurate and robust
pupil detection, but for now a simple detector will do. Later I will revisit the detection when the
basics of the other components are done.</p>
<p>The idea presented here is a simple one. If we look at the image, the pupil is just a black circle.
Now this might not always be true, as in the circle does not always have to be completely black (in
case we have some glints) and might not be a circle (if we're looking at the pupil under an angle).
We will deal with these potential issues later, for now we'll just stick to the basics.</p>
<p>I will be using <a href="http://opencv.org/">OpenCV 2</a> and <a href="http://www.python.org/">Python 2.7.5</a> on OSX.</p>
<p>So on a high level, the process is the following:</p>
<ul>
<li>Load the image and convert it to grayscale</li>
<li>Threshold the image</li>
<li>Cleanup using closing</li>
<li>Find the most circular contour</li>
<li>Draw it into the original image</li>
</ul>
<p>So here I just load the original image.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">import </span><span>cv2
</span><span style="color:#b48ead;">import </span><span>numpy </span><span style="color:#b48ead;">as </span><span>np
</span><span>
</span><span>image = cv2.</span><span style="color:#bf616a;">imread</span><span>('</span><span style="color:#a3be8c;">eye.png</span><span>')
</span><span>gray = cv2.</span><span style="color:#bf616a;">cvtColor</span><span>(image, cv2.cv.</span><span style="color:#bf616a;">CV_BGR2GRAY</span><span>)
</span></code></pre>
<p><img src="https://mirosval.sk/blog/2014/pupil-detection/001.png" alt="Eye image" /></p>
<p>Now that that's out of the way, I'll threshold it. This will make a binary image out of a grayscale
one. So everything darker than 30 goes to black, and everything else goes to white. And this is what
I get.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>retval, thresholded = cv2.</span><span style="color:#bf616a;">threshold</span><span>(gray, </span><span style="color:#d08770;">30</span><span>, </span><span style="color:#d08770;">255</span><span>, cv2.cv.</span><span style="color:#bf616a;">CV_THRESH_BINARY</span><span>)
</span></code></pre>
<p><img src="https://mirosval.sk/blog/2014/pupil-detection/002.png" alt="Threshold image" /></p>
<p>Now this next part (Line 1 and 2) does not have to be here for this particular image, but in general
it should be there, becuase you want to get rid of small holes that might occur in the pupil from glints,
but since there are none in this image, it is of no concern.</p>
<p>Now the binary image is fed into the
<a href="http://docs.opencv.org/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html?#findcontours">findContours</a>
function, which will identify individual "objects"
in the image. In this case we get two, one is for the pupil and one is for the whole image. So how to
distinguish between the two? I used the <code>extend</code> parameter which puts in relation the area of a shape
to the area of its bounding box. This will be close to 1 for the whole image, because it is rectangular,
but lower for the pupil, since it is more circular. Something like
<a href="http://docs.opencv.org/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html?#convexhull">Convex Hull</a> and
<a href="http://docs.opencv.org/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html?#arclength">Arc Length</a>
could be used to make this more robust. For example use the ratio of the radius to area. I will
revisit this topic in the future.</p>
<p>At last we compute the center of the detected area and fit an ellipse around it, so that we can
draw the results into the last image.</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span>kernel = cv2.</span><span style="color:#bf616a;">getStructuringElement</span><span>(cv2.</span><span style="color:#bf616a;">MORPH_RECT</span><span>, (</span><span style="color:#d08770;">10</span><span>, </span><span style="color:#d08770;">10</span><span>))
</span><span>closed = cv2.</span><span style="color:#bf616a;">erode</span><span>(cv2.</span><span style="color:#bf616a;">dilate</span><span>(thresholded, kernel, </span><span style="color:#bf616a;">iterations</span><span>=</span><span style="color:#d08770;">1</span><span>), kernel, </span><span style="color:#bf616a;">iterations</span><span>=</span><span style="color:#d08770;">1</span><span>)
</span><span>
</span><span>contours, hierarchy = cv2.</span><span style="color:#bf616a;">findContours</span><span>(closed, cv2.cv.</span><span style="color:#bf616a;">CV_RETR_LIST</span><span>, cv2.cv.</span><span style="color:#bf616a;">CV_CHAIN_APPROX_NONE</span><span>)
</span><span>
</span><span>drawing = np.</span><span style="color:#bf616a;">copy</span><span>(image)
</span><span>
</span><span style="color:#b48ead;">for </span><span>contour </span><span style="color:#b48ead;">in </span><span>contours:
</span><span> area = cv2.</span><span style="color:#bf616a;">contourArea</span><span>(contour)
</span><span> bounding_box = cv2.</span><span style="color:#bf616a;">boundingRect</span><span>(contour)
</span><span>
</span><span> extend = area / (bounding_box[</span><span style="color:#d08770;">2</span><span>] * bounding_box[</span><span style="color:#d08770;">3</span><span>])
</span><span>
</span><span> </span><span style="color:#65737e;"># reject the contours with big extend
</span><span> </span><span style="color:#b48ead;">if </span><span>extend > </span><span style="color:#d08770;">0.8</span><span>:
</span><span> </span><span style="color:#b48ead;">continue
</span><span>
</span><span> </span><span style="color:#65737e;"># calculate countour center and draw a dot there
</span><span> m = cv2.</span><span style="color:#bf616a;">moments</span><span>(contour)
</span><span> </span><span style="color:#b48ead;">if </span><span>m['</span><span style="color:#a3be8c;">m00</span><span>'] != </span><span style="color:#d08770;">0</span><span>:
</span><span> center = (</span><span style="color:#bf616a;">int</span><span>(m['</span><span style="color:#a3be8c;">m10</span><span>'] / m['</span><span style="color:#a3be8c;">m00</span><span>']), </span><span style="color:#bf616a;">int</span><span>(m['</span><span style="color:#a3be8c;">m01</span><span>'] / m['</span><span style="color:#a3be8c;">m00</span><span>']))
</span><span> cv2.</span><span style="color:#bf616a;">circle</span><span>(drawing, center, </span><span style="color:#d08770;">3</span><span>, (</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">0</span><span>), -</span><span style="color:#d08770;">1</span><span>)
</span><span>
</span><span> </span><span style="color:#65737e;"># fit an ellipse around the contour and draw it into the image
</span><span> ellipse = cv2.</span><span style="color:#bf616a;">fitEllipse</span><span>(contour)
</span><span> cv2.</span><span style="color:#bf616a;">ellipse</span><span>(drawing, </span><span style="color:#bf616a;">box</span><span>=ellipse, </span><span style="color:#bf616a;">color</span><span>=(</span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">0</span><span>))
</span><span>
</span><span>plt.</span><span style="color:#bf616a;">figure</span><span>(</span><span style="color:#bf616a;">figsize</span><span>=(</span><span style="color:#d08770;">10</span><span>, </span><span style="color:#d08770;">5</span><span>))
</span><span>plt.</span><span style="color:#bf616a;">imshow</span><span>(drawing)
</span></code></pre>
<p><img src="https://mirosval.sk/blog/2014/pupil-detection/003.png" alt="Final image" /></p>
<p>So that is how we quickly detect the position of a pupil in an image. In the next installments
we will look at how to detect the iris, which is a bit more complicated. We will also flatten
the iris out and then we will look at processing video.</p>
<p>This post is also available as a <a href="http://nbviewer.ipython.org/gist/mirosval/8752402">iPython Notebook</a></p>
Intro2014-01-28T00:00:00+00:002014-01-28T00:00:00+00:00https://mirosval.sk/blog/2014/intro/<p>Hi everyone, I intend to use this blog thing to write about the progress of my Master Thesis.
I'll be talking about implementing various computer vision algorithms together to form an
eye-tracker. In between I'll talk about other things if something interesting comes up.</p>