State of the ecosystem of Common Lisp for 2015

From the translator:
I've heard that Common Lisp libraries from the ' 80s, and others just do not, but many of its users is a 3.5 professors, working on artificial intelligence DARPA in the catacombs and do not have even close idea about the daily tasks of the average programmer. This article shows well that although you may stumble upon an old library, but use is only modern counterparts, that existing libraries are constantly evolving and new ones appear.

The author of this article, Fernando Borretti is the active contributors to the ecosystem of Common Lisp, author of more than 30 libraries, most of which are for web development.

Passing by the reader will get a General idea of the situation in Common Lisp, investigators will be able to understand that he needs to try to write the code and which libraries to take for a test task, and an experienced developer learns about the latest developments, understand in what areas he can help the community and get some tips on how to answer questions, not to kill their interest in this wonderful technology. Go!




Here is a description of ecosystem Common Lisp for August 2015 from the point of view of the user and contributor.

The purpose of this article is to give an overview of the ecosystem and to help introduce unification in each area.

For each application area listed the recommendations for the unification of this part of the ecosystem, as well as interesting directions for future development.

the

application Development



the

Web development



the

the Backend



Clack, the analogue of WSGI/Rack exists since 2009 and is well tested and running in production. The three web framework — Caveman2, Ningle and Lucerne created based on it.

Clack is an abstraction of the HTTP server, which allows the user to write web frameworks (or better frameworks web applications) without ties to a specific server implementation.

The importance of using Clack cannot be underestimated: if you make the application directly, say, Hunchentoot, Hunchentoot you linked to, and if there is a new, faster server, like Woo, you will have to rewrite whole application to use it. If you write a plugin for Clack like clack-errors, it can automatically be used by all applications, regardless of the frameworks made in Clack, reducing needless duplication of code.

With Clack switch with Hunchentoot on Woo and joy from the stunning acceleration it's just a matter of setting libev and change one key argument.

Conclusion:

Stop to use Hunchentoot directly. Use the Clack, or even better, one of the frameworks based on it.

Future work:
The main part is finished, it is time to write high-level stuff. Extensible framework administration Clack apps like Django is a good example.

the

Frontend



It all has to do with the ability of Common Lisp compiled to JavaScript. Common Lisp is quite a large language, and a bit of options:

the
    the
  • Parenscript: DSL which compiles a subset of Common Lisp to idiomatic JavaScript
  • the
  • JSCL: samoraspustitsya the compiler from CL to JS. There is no support for CLOS, format, and loop. (unlike Parenscript is an interpreter in the browser — approx. interpreter)


Conclusion:

The best way to unify is the development of one of the existing CL implementations in JS.

Future work:

Something like CFFI, but implementation of CL in JS, so you don't have to rewrite binding to the JavaScript libraries with new versions of the libraries.
Utility to compile Common Lisp to JavaScript are independent of specific implementations to ease the transition from Parenscript to JSCL, or other implementations.

the

Command line



For many years various tools have appeared in this area, the last of them, the one that had the biggest momentum — Roswell — implementation Manager/installer and launcher scripts. One of the great features — easy to compile small scripts into executable files, for example generating the documentation.

It is also used to install implementations (CL) in Travis to bypass some of the problems with cl-travis.

Conclusion:

Kill cl-launch, use Roswell.

Future work:

More Roswell scripts.

the

Graphical interface



For a long time a big concern was the lack of a complete solution for GUI. Now we have it: CommonQt plus Qtools. The first years used in real applications and the second layer, allowing you to do everything easier.

The biggest problem in using CommonQt is a job requirement Smoke, and to make the libraries work can be difficult, especially in a system other than Linux. This is achieved thanks to the Qtools, which depends on a library called qt-libs. It downloads the Smoke library for the current platform and makes setting up and deploying applications much easier.

Conclusion:

To focus on CommonQt and to help improve cl-cffi-gtk and other libraries considered obsolete.

CLIM interesting and was the last attempt to study ways of creating user interfaces, but it is not a viable option in 2015.

Future work:

More tutorials and examples of using CommonQt and Qtools.

the

Machine learning



CLML pretty versatile solution. It was developed Mathematical Systems Inc., a Japanese company. Mike Maul later moved it to github and cleaned a little code. Available tutorial on time series.

Another candidate in this region is mgl used the author to win the Higgs Boson Machine Learning Challenge.

In the field of numerical computing I have always been interested library Antik, but unfortunately it depends on GNU Scientific Library that makes it be distributed under GPL. As is mgl-mat and LLA.

Conclusion:

Combining pieces for the calculations of the LLA and mgl-mat and parts of machine learning from CLML and mgl will solve most of the problems in this area.

Future work:

CLML probably contains a lot of code for calculations which can be placed in a separate library-type SciPy and NumPy.

the

Databases



cl-dbi provides a generic interface to various database-specific libraries (cl-postgres, cl-mysql, etc.). SxQL gives you a DSL for constructing safe, automatically generic SQL queries.

There is two equals a full-fledged ORM: Crane from the author of this review and Integral from the author cl-dbi.

Conclusion:

To discourage the use of anything other than cl-dbi.
Future work:

Binding for other databases, such as Oracle, exist. Writing drivers for cl-dbi will be the best way to promote and aid harmonization.

the

Graphics



I've never programmed schedule, so I have little knowledge in this area. There are CEPL related project Varjo, with excellent collecta video tutorials. Of course, there are low level libraries like cl-opengl and cl-sdl2.

Conclusion:

To promote the use of CEPL, because it is almost full.

Future work:

High-level OpenGL library, like pg would be great.

More libraries in this field, especially for handling a variety of meshes and other 3D formats.

the

Competition



cl-async probably the most complete solution for everything related to competition. And it is built on top of libuv, libraries, behind Node.js.

Other notable libraries in this area:

the
    the
  • STMX: provides support for software transactional memory, and very impressive.
  • the
  • lparallel: a complete framework for parallel programming.


Library type legion facilitate competitiveness for certain cases.

Conclusion:

Future work:

In this area a lot of space for new ideas.

the

file Formats



In Common Lisp has libraries for all popular file formats:

the

The new JSON libraries Jonathan, very fast JSON generator and parser.

Conclusion:

There are too many XML and JSON libraries, it leads to paralysis of choice.

Future work:

YAML parser to cl-yaml did not depend on libyaml, it will make distribution much easier.

the

Interaction



UIOP and the level of compatibility of ASDF contain a huge set of tools for porting any functionality from the request host name to start external programs and manipulation of environment variables.

Conclusion:

Use UIOP for everything. Do not use the cl-fad. Do not use OSICAT.

Future work:

An interesting project in this area would be a DSL for generating LLVM IR, or string, or using binding LLVM API. This would allow to use Common Lisp as the Assembly language with an incredibly powerful macro system. For example, to JIT-compile to highly optimized SIMD vectorized numerical code.

the

Miscellaneous



the

Benchmarks



The creation of a framework like Criterion in Haskell it would be a big step forward.

the

Parse



Then I usually choose esrap, which generates parsers from Lisp descriptions of grammars. Need more diversity in this area. Recently there was a release proc-parse, fast library for parsing the vector.

the

Logging



Not much you can say about it, except that Common Lisp has good utilities for logging, such as log4cl and vom.

the

Development



the

type System



Again, there is nothing to say, except that Common Lisp is a pretty good type system, which is not used to its fullest.

Conclusion:

CLOS have no competition among the surviving versions of Flavors, and it's practically the only OOP implementation in Common Lisp.
Future work:

I often look at trivial types, a library that contains many simple type specifiers. Library more, including CDR, would be good.

the

Testing



There are many frameworks available for testing, the main of them FiveAM and the newer Prove.

Of the existing library and Prove to be a good candidate. Extensible report generators a great thing, and a common approach to extensibility makes it a good solution to the problem of test frameworks.

Other, older libraries depend on frameworks such as rt or lisp-unit, but they are no longer capable.

Conclusion:

To prevent the use of test frameworks in addition to FiveAM and Prove.

Future work:

Write new test frameworks would be counter productive in the current situation. Work should be focused on harmonization of existing ones.

the

Testing in the cloud



Common Lisp has good support for the use of services like Travis and Coveralls to test your code and traceability in the cloud, respectively. The two main libraries for this cl-travis and cl-coveralls and their both very easy to use. In my experience testing and tracking coverage of the project in Common Lisp is definitely easier than the same project in Python.

Conclusion:

cl-travis has a few problems, for example, it requires sudo to install implementations, because of this he can't use the new Docker-based infrastructure on Travis.

Roswell solves this problem, so it is recommended to use Roswell with cl-travis. Here you can find the tutorial for using Roswell with Travis.

Future work:

A lot of services such as Circle CI, is always useful. Tutorials will also be a good addition.

the

Documentation



For online, automatically updating documentation style Read the Docs is Quickdocs, which gets the description of the API project by using docparser.

There are surprisingly few documentation generators. I use the Codex for documentation generation. It is written in CommonDoc — library that provides format-independent internal representation of documents. Codex is conceived similar to the Sphinx: documentation written prose, and you insert the automatically extracted API documentation (functions, macros, classes, and so on) in text using macros.

Conclusion:

To promote the use Quickdocs referring to it from the README, advising new users etc.

Future work:

I will continue work on the Codex and CommonDoc, and any improvement in CommonDoc (mainly with new output formats and macros) will be reflected in the Codex.

the

Unicode



It's essentially a solved problem for most implementations. Sometimes you may run into a problem if you have installed some kind of UTF-8 encoding, but it is easily corrected, for example in SBCL:

the
(setf sb-impl::*default-external-format* :utf-8)


I've never met other problems with Unicode, even when interacted with the libraries in C?

Conclusion:

Some implementations are lagging behind in support for Unicode, such as ABCL.

Future work:

To eliminate issues in the Unicode support can improve the library cl-unicode.

the

Managing packages



Quicklisp de facto package Manager for Common Lisp. And that is enough. Don't ruin something that works.

Conclusion:

All good.

Future work:

Utilities based on Quicklisp, Quickdocs and qlot.
the

build System



ASDF, the de facto build system for Common Lisp. Everyone's using her.

Each project has a file .asd, it is called a system definition specifies the project metadata (author, maintainer, homepage, etc) and components.

For me, this is one of the main attractive features of Common Lisp. In languages like Python, each file is importing what they need, and the project becomes a massive count of cross-file dependencies. In ASDF you simply list your project files in the order in which they are defined. Or you can specify dependencies between files and allow ASDF to calculate the procedure itself. The point is that dependencies are specified in clearly and distinctly visible.

But enough agitation. The important thing is that ASDF has no competitors, not because no one wants to create an analogue, but because ASDF has destroyed all alternatives a few years ago. And that's good.

Conclusion:

All good.

Future work:

More components to the ASDF, for example, to build C/C++ files. Platform-independent package Manager for loading external C libraries required libraries Lisp would be incredibly useful.

the

development Environment



SLIME is a development environment for Common Lisp-based Emacs and most widely-spread. The fact that it is made in Emacs is a problem, because to tell new users that they have to learn Emacs to use SLIME (which wasn't entirely true) is a significant artificial barrier to the study of CL.

Conclusion:

To reduce the entry barrier for using SLIME.

Future work:

Environment different from Emacs can't hurt. Plugin for Atom, who worked with Swank-server would be very useful.

the

Conclusion



the

Specific problems



the

Communication



The number of people creating web applications in Common Lisp and do not know about the Clack is too large. It is obvious that there is a problem in the inform potential users about good libraries.

the

the Paralysis of choice



When someone asks what library to use for writing code in a particular area should be recommended only the best library of this region. To say "you can take X, Y and Z" only worsens the paradox of choice.

If someone asks the library for the graphical interface, answer "CommonQt Qtools", don't add "but you can also take cl-cffi-gtk or LTK..."

If you ask what to use for web apps, share a link to one of the frameworks Clack. Not talking about Hunchentoot Wookie Woo or or or any other server.

If someone asks what kind of development environment to use — specify them on the SLIME. Don't tell them about some shady abandoned project of 2005 which only works with CLISP.

And most importantly: when someone asks you what implementation to use — just answer SBCL or CCL. Don't say "well, you can take ECL for embedding or ABCL to run on the JVM". They are to be used in more narrow cases, not in order to give a quick start for the beginner.

the

Development



the

Downloads from Quicklisp



Below are the total number of downloads for the hundred most popular projects of the Quicklisp, between January and July:



the

Repositories



I signed up for the tape the new repositories for Common Lisp on Github. I also wrote a bit of code to query database Newsbeuter:

the
(ql:quickload (list :dbi :yason :local-time :group-by))

(defvar *connection*
(dbi:connect :sqlite3
:database-name (merge-pathnames #p".newsbeuter/cache.db"


(let* ((output (merge-pathnames #p"lisp github.json"
(user-homedir-pathname)))
(query (dbi:prepare *connection*
"SELECT pubDate FROM rss_item WHERE feedurl = ?"))
(results (mapcar #'(lambda (result)
(local-time:unix-to-timestamp
(getf result :|pubDate|)))
(dbi:fetch-all
(dbi:execute query "http://planet.lisp.org/github.atom")))))
(with-open-file (stream output
:direction :output
:if-exists :supersede
:if-does-not-exist :create)
(yason:encode
(mapcar #'(lambda (group)
(list
(local-time:format-timestring nil
(first group)
:format (list :year #\- :month #\- :day))
(1- (length group))))
(group-by:group-by-repeated results
:keys (list #'identity)
:tests (list #'(lambda (a b)
(= (local-time:day-of a)
(local-time:day-of b))))))
stream)))



From 23 Mar 2015 to 20 August 2015 was created 882 repository.



the

Acknowledgement



Thank you Javier Olaechea for his comments about Unicode support, Eitaro Fukamachi and François-René Rideau, Gabriel Gonzalez for the original article on the state of the Haskell ecosystem.
Article based on information from habrahabr.ru

Комментарии

Популярные сообщения из этого блога

Briefly on how to make your Qt geoservice plugin

Database replication PostgreSQL-based SymmetricDS

Yandex.Widget + adjustIFrameHeight + MooTools