Geospatial Query with MongoDB and Node.js

Hasn’t been a long time when i started working with Wearetech.io and my first challenge was on. Sweet!

If you want to learn more about Wearetech.io, check the site and our Github repo.

Now on the fun part. We have a use case where an individual would be able to register on our site as a City Curator.

He could search for his City and if he wouldn’t find it in our database he would register it. When the next candidate comes in, he searches for his place and if it would be inside a 10 kilometers range of an already registered city we would deny it, since we wouldn’t want to have city calendars overlapping with each other.

When Thanasis started building Wearetech.io he decided to go with MongoDB. So in order to create the above scenario, MongoDB Geospatial Queries to the rescue!

MongoDB supports two types of Geospatial Queries indexing.

  • The spherical (2dSphere), that would allow you to store shapes made of points (coordinates) and then make comparisons with them like intersection, etc.
  • The flat (2d), that would store single points and then come up with their distances.

Your choice will affect your development a lot down the road, so think wisely.

Now on the code side, we will use Mongoose to shape our models. Here’s the most simple one.


var mongoose     = require('mongoose');
var Schema     = mongoose.Schema;

var CitySchema   = new Schema({
  name: String,
  geo: {
    type: [Number],
    index: '2d'
  }
});

module.exports = mongoose.model('City', CitySchema);

Now we can start populating our DB like this


var City = require('./app/models/city');

var cityModel     = new City(); 
cityModel.name = req.body.name; 
cityModel.geo    = [ req.body.lat, req.body.lng ]; 

cityModel.save(function (err) {
  if (err)
    res.send(err);

  res.json({});
});

And now on the fun part

var distance = 1000 / 6371;

var query = City.findOne({'geo': {
  $near: [
    req.body.lat,
    req.body.lng
  ],
  $maxDistance: distance

  }
});

query.exec(function (err, city) {
  if (err) {
    console.log(err);
    throw err;
  }

  if (!city) {
    res.json({});
  } else {
    console.log('Cant save: Found city:' + city);
    res.json(city);
 }

});

Our distance is in radians metric system and you can find more on how to calculate it here.

Here is a demo of it. Try searching your City. If it’s available register it and then try to find a place inside a 10 kilometers range of your previeus registered City.

Hope you will find it interesting. Comments?

How To Be A Programmer by Robert L Read

http://samizdat.mines.edu/howto/HowToBeAProgrammer.html

Recently, my colleague Yannis Rizos, tweeted me about a book. Its title was somehow offensive and didn’t look quite interesting at first.

But once i started browsing it, i catch some really good points and decided to start from the beginning.

Written by Robert L Read, it’s a free essay, ~100 pages long.

It is listing all the borrows and challenges a programmer will face and must pass in order to become a programmer.

Starts with the most essential skill of a programmer, debugging. How to debug and some of the most trivial bugs you will eventually find in your code.

It goes on with more skills like motivation, evolution, how to stress test, be a team player and communicate effectively with every type of people involved in this industry.

It is divided in 3 chapters, Beginner, Intermediate and Advanced and every chapter contains a few paragraphs for every concept.

I was really amazed by this essay and i would surely recommended to every beginner, junior, mid-senior professional programmer. I guess senior would already know these stuffs.

What are you favorite recent reads?

Valiation : The most awesome validation engine ever created for PHP

https://github.com/Respect/Validation

The past few weeks, you would find Validation at Github’s PHP Trending projects.

Validation is a very flexible, fully customizable, loaded with tone of validators, engine that you can use on your PHP projects right away.

Here is a great list why this library it’s actually awesome.

From Reddit:

  • The chain is not a simple chain (it is not just linear), it is a fluent builder for a composite structure. You can write almost any validation rule for any data structure and group it in a single object:v::key(“name”, v::string()->length(1, 32)) ->key(“birth”, v::date(‘Y-m-d’)->minimumAge(18)) ->assert($someArrayWithData);
  • You can nest as many validators as you want.
  • Each validator is an instance that you can reuse (even for composing new, more complex instances).
  • Three kinds of validation reports (validate() returns true/false, check() stops on first error, assert() collect all errors).
  • Nested reports implement RecursiveIteratorIterator AND RecursiveTreeIterator (that’s where the ASCII tree report came from!)
  • A selector API for finding messages in complex nested reports (findMessages([“user.address.street.length”])).
  • Reports are only generated when needed (true/false validation doesn’t even touch the reporting system).
  • Really easy to extend (most rules have a single method).
  • Really easy to make inline rules:v::callback(function ($input) { return $input == 42; });
  • Logic operations on any validator:v::allOf(v::numeric(), v::hexa(), v::min(1)); // numeric, hexadecimal and at least 1 v::oneOf(v::nullValue(), v::numeric()); // null or numeric
  • Integrates with ZF1, ZF2 and Symfony2 validators if needed. Dependency is optional.
  • A full concrete API (not relying on magic methods or chains) that can be built using dependency injection.
  • Straightforward to use on unit tests instead of the PHPUnit assertion library.

Here i wrote a simple User Model class where we can make our validation:

Validation will throw an exception when it fail and you can inform your users.

Maintaining a Magento code base

Magento is hard. No doubt about it. But this is not an excuse. Beside that (and a few other issues) it’s still the number one e-commerce solution. It’s proven.

So, i wanted to talk a little about how i maintaining my Magento projects.

1) I am keeping everything under version control

Magento’s folder structure brings a lot of frustration. An extension can place it’s files almost anywhere it’s developer wants. It can be in base package theme either in package default theme. It can be in media folder for permanent files you want to keep under control either in /js folder for Javascript global libraries.

Keeping a .gitignore with all the Project’s specific files is almost impossible.

Also, you can’t carry with you your Project’s modules, except using Composer with Magento which it’s also complicate to work with. (Still struggling with it on Windows)

So, what i do is keep everything under version control, except temporary files like cache, logs, etc. etc.

2) Never touch core files

By never touching core files like in folders:

  • app/code/core
  • app/locale/
  • app/design/frontend/(base|default|rwd)
  • app/design/adminhtml/
  • js/ (initial folders only)
  • lib/ (initial folders only)
  • skin/frontend/(base|default|rwd)
  • includes/

you make sure that, when a feature release is out, you can just copy paste above you project and everything still works fine.

Keeping your Magento always up to date can save you from big disasters. I must make sure that feature releases wont be a pain to get.

3) Extend everything

In case you need to extend a Magento’s core functionality or make it behave different, you are covered.

Magento’s team had a purpose and only one. To make the most extensible e-commerce framework out there. They even sacrifice performance on this purpose. Now, some may argue that that was stupid or they didn’t success after all. I will have to disagree with both.

Magento is a true PHP OOP project, powered by the most powerful framework, the Zend Framework.

As about performance, these days you have a lot of tools in order to create a fast website. You may be hosted at an AWS super server, use fast caching methods, etc, etc.

4) Keep a local Magento with stuff i use a lot

After a few projects, i realized that a lot of processes were repeated again and again. The same modules i will installed on every fresh Magento install, the same configuration i would do. So i keep a private(yet!) Magento repository were i have all of my Modules and a clean Database dump with my configuration set.

How do you deal with your Magento projects? Share your tactics and technics with me on Twitter or bellow in the comments.

Add configuration file to your PHP Projects

https://github.com/vlucas/phpdotenv

For a while now, i’ve been using Symfony’s YAML to add a configuration file on my PHP projects.

Today, i stumbled upon Vance Lucas’s PHPDotEnv which is a great PHP package that will transfer the contents of a .env file right inside your PHP’s $_ENV variable. Available through Composer too.

Easiest method to add configuration files to your PHP projects ever!

Magento(Varien lib) & PHP memory_limit -1 bug

Magento

Last week an unusual bug really got me some time until i figured out.

Common symptoms are Magento not creating your product’s images cache without a reason or when trying to upload product’s image and an error says ‘Memory limit reached’.

A lot of resources on the Internet says that you have to increase your PHP’s memory limit. But when i checked my PHP’s configuration i noticed that memory limit was to -1, which means that a script is allowed to allocate all your available memory. Hmm, that’s interesting. Hmmm, that’s interesting.

I followed the stacktrace and i found this (lib/Varien/Image/Adapter/Gd2.php):


protected function _isMemoryLimitReached()
{
  $limit = $this->_convertToByte(ini_get('memory_limit'));
  $size = getimagesize($this->_fileName);
  $requiredMemory = $size[0] * $size[1] * 3;
  return (memory_get_usage(true) + $requiredMemory) > $limit;
}

[...]

protected function _convertToByte($memoryValue)
{
  if (stripos($memoryValue, 'M') !== false) {
    return (int)$memoryValue * 1024 * 1024;
  } elseif (stripos($memoryValue, 'KB') !== false) {
    return (int)$memoryValue * 1024;
  }
  return (int)$memoryValue;
}

There you have it. The author here doesn’t check for when memory_limit is set to -1. Instead he assumes that it will be always set to form of XXM or XXKB.

So in case you have bumped into the above symptoms check out that your PHP’s memory limit is not set to infinite.