Long time ago (12 years) there was a discussion about hashtables
. Jon Harrop stated that Haskell cannot simply have fast mutable hashtables. The fastest hashtables in Haskell were slower than F# (.NET Generic dictionaries), OCaml (Hashtbl
) and even Python implementations.
The main reason of such slowness in benchmarks across implementations was in unboxed ints for keys and values that were used for .NET Generic dictionaries while Haskell HashMap
used boxed ints.
Two years later Gregory Collins introduced hashtables
package with different hashtables provided via type class HashTable h
. At the same time, it also had and still has performance issues.
@A64mQ
who also known as @klapaucius and @A64m_qb0
was tired of it and made proof of concept of effecient mutable hashtables based on vectors. He also provided benchmark that demonstrates comparison between vector-hashtable
and hashtables
. Vectors of hashcodes, references, buckets and metadata had been replaced with primitive arrays later.
Results for GHC 9.0.1 could be found here:
cabal bench --benchmark-options="--output $HOME/Downloads/vhc.html Comparison" \
+RTS -N4 -A64m -n4m -qb0 -RTS
cabal bench --benchmark-options="--output $HOME/Downloads/vhu.html Utilities" \
+RTS -N4 -A64m -n4m -qb0 -RTS
One year ago I joined him to push it till release on Hackage.
Hashtables are a data structure that represents associative array of entries.
It means that usually hashtables allow constant O(1)
lookup, insertions and modifications. In case of collisions buckets should be traversed to get the index of exact entry. In the worst case it’s O(n)
.
Here, in vector-hashtables
open addressing approach was used.
It could be described in following timeline in case you are wondering.
2020-08-08
: “project kick-off” for me.2020-08-12
: tests were added.2020-10-01
: utilities were added.2021-01-16
: utilities performance fixed.2021-08-18
separate Utilities
benchmark created, toList
and fromList
functions added to Comparison
benchmark.2021-08-22
Int
mask support for 32-bit architecture added.2021-09-01
haddocks provided.2021-09-07
Hackage release: https://hackage.haskell.org/package/vector-hashtables-0.1.0.1Despite of the challenges I met among the way, we finally get it done.
@A64m_qb0
for creating data structure, benchmarks and accepting my support.I am using servant
to serve dynamic websites, as well as APIs. Last time I added SEO plugin to its ecosystem. This time I am going to extend servant
itself. Here we go.
Consider URI fragment. In Web it could be recognized as HTML anchor. It is not part of REST API. It is a part of URL. And not the deprecated one.
Consider GitHub, e.g. follow the link and look on its URL: https://github.com/haskell-servant/servant/blob/master/default.nix#L22. Page content is the same with and without anchor #L22
at the end of URL. This anchor changes content representation (i.e. highlights line #22
).
I would like to have this feature available inside servant
core package, since fragment is a part of URI and could be used on both server and client side. Despite it might be abused in non-HTML use-cases.
According to RFC 3986, Fragment provides secondary information in advance to primary URI. It might be specified or not. It could be at the end of URI.
It could be appended directly to path and only once. Otherwise, if query string is not empty, it could be present at the end of query string.
According to RFC 2616, Fragment should be secured on both client and server sides. curl
strips the fragment from URL. Web browsers also removes it before sending to server.
Nevertheless, fragment could be served in redirects on server side. It could appear on HTML page as an anchor to corresponding section and even used for indexing single page applications.
Like sitemap, it is supported in yesod
framework but not in servant
.
data Fragment (a :: *)
that transformed to Maybe a
on both client and server side.However, it turns out that is not quite good to decide about parsing behaviour for users.
data Fragment' (mods :: [*]) (a :: *)
.During discussion about implementation details with Servant Contributors I was convinced to follow HTTP spec and remove client/server part of the feature.
Thus, the final approach looks like:
data Fragment (a :: *)
.Link
.servant
is a single repository with multiple packages inside. It represents the core of servant ecosystem and maintained by Servant Contributors.
It provides following packages:
servant
(core).servant-client-core
.servant-client
.servant-server
.servant-docs
.servant-foreign
.All requirements will be splitted across the packages.
servant
Servant.Links
should support fragment.Fragment combinator defined as unhabitant data type (requirements #1, #8).
data Fragment (a :: *)
The main idea here is to produce type-safe links for specific endpoints within API.
There is a Link
.
class HasLink endpoint where
type MkLink endpoint (a :: *)
toLink
:: (Link -> a)
-> Proxy endpoint -- ^ The API endpoint you would like to point to
-> Link
-> MkLink endpoint a
HasLink
is the way to make the Link
through the API from its pieces.linkURI'
and linkURI
are the functions to produce target URI
from Link
.safeLink'
and allLinks'
.And also instance ToHttpApiData Link
is also affected.
Since fragment is optional part of URI and it could be present separately from path or query, Link
should be extended with fragment:
data Link = Link
{ _segments :: [Escaped]
, _queryParams :: [Param]
, _fragment :: Maybe String
} deriving Show
Fragment should become a member of HasLink
type class (requirement #9):
instance (HasLink sub, ToHttpApiData v)
=> HasLink (Fragment v :> sub) where
type MkLink (Fragment v :> sub) a = v -> MkLink sub a
toLink toA _ l mv =
toLink toA (Proxy :: Proxy sub) $
addFragment ((Just . Text.unpack . toQueryParam) mv) l
MkLink
type expands user supplied type to a function parameter.Link
would be extended with a Fragment
via addFragment
helper.URI
would be updated with escaped fragment (requirements #9, #10):
linkURI' :: LinkArrayElementStyle -> Link -> URI
linkURI' addBrackets (Link segments q_params mfragment) =
URI mempty -- No scheme (relative)
Nothing -- Or authority (relative)
(intercalate "/" $ map getEscaped segments)
(makeQueries q_params)
(makeFragment mfragment)
where
makeQueries :: [Param] -> String
makeQueries [] = ""
makeQueries xs =
"?" <> intercalate "&" (fmap makeQuery xs)
makeQuery :: Param -> String
makeQuery (ArrayElemParam k v) = escape k <> style <> escape (Text.unpack v)
makeQuery (SingleParam k v) = escape k <> "=" <> escape (Text.unpack v)
makeQuery (FlagParam k) = escape k
makeFragment :: Fragment' -> String
makeFragment Nothing = ""
makeFragment (Just fr) = "#" <> escape fr
style = case addBrackets of
LinkArrayElementBracket -> "[]="
LinkArrayElementPlain -> "="
Link
instance would become extended with fragment.
instance ToHttpApiData Link where
toHeader = TE.encodeUtf8 . toUrlPiece
toUrlPiece l =
let uri = linkURI l
in Text.pack $ uriPath uri ++ uriQuery uri
in Text.pack $ uriPath uri ++ uriQuery uri ++ uriFragment uri
Servant offers several type families to operate with API on type level.
Firstly, I wanted to practice with them.
>>> :set -XDataKinds -XTypeOperators -XPolyKinds -XTypeFamilies
>>> :kind! Endpoints (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent)
Endpoints (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent) :: [*]
= '[Fragment Bool
:> (Fragment Int :> Verb 'GET 200 '[JSON] NoContent)]
>>> :{
| type family Filter (a :: *) (bs :: [*]) :: [*] where
| Filter a '[] = '[]
| Filter a (a ': bs) = a ': Filter a bs
| Filter a (b ': bs) = Filter a bs
| :}
>>> :kind! Filter Bool '[Int, Char]
Filter Bool '[Int, Char] :: [*]
= '[]
>>> :kind! Filter Bool '[Int, Char, Bool, String]
Filter Bool '[Int, Char, Bool, String] :: [*]
= '[Bool]
Then I came with following algorithm: Map Snd (Filter ((> 1) . Fst) (Map (CountInEndPoint Fragment) (Endpoints api))
.
Endpoints
.However, I did not succeeded with this approach. Idea of Map
as higher-order type-family that could accept type family as an argument and apply it to every list element was a bit hard for me.
And then it clicked. Constraint! And again: I do not need a list of types!
Type error I want to produce:
>>> :{
| type NotUniqueFragmentInApi api =
| 'Text "Only one Fragment allowed per endpoint in api ‘"
| ':<>: 'ShowType api
| ':<>: 'Text "’."
| :}
>>> :kind! (NotUniqueFragmentInApi (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent))
(NotUniqueFragmentInApi (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent)) :: ErrorMessage
= NotUniqueFragmentInApi
(Fragment Bool :> (Fragment Int :> Get '[JSON] NoContent))
Type family for checking that fragment should not be present in API:
>>> :{
| type family FragmentNotIn api orig :: Constraint where
| FragmentNotIn (sa :<|> sb) orig =
| And (FragmentNotIn sa orig) (FragmentNotIn sb orig)
| FragmentNotIn (Fragment c :> sa) orig = TypeError (NotUniqueFragmentInApi orig)
| FragmentNotIn (x :> sa) orig = FragmentNotIn sa orig
| FragmentNotIn (Fragment c) orig = TypeError (NotUniqueFragmentInApi orig)
| FragmentNotIn x orig = ()
| :}
<interactive>:45:3: error:
• Illegal nested type family application ‘And
(FragmentNotIn sa orig) (FragmentNotIn sb orig)’
(Use UndecidableInstances to permit this)
• In the equations for closed type family ‘FragmentNotIn’
In the type family declaration for ‘FragmentNotIn’
GHC advises to enable UndecidableInstances
extension. OK!
>>> :set -XUndecidableInstances
>>> :{
| type family FragmentNotIn api orig :: Constraint where
| FragmentNotIn (sa :<|> sb) orig =
| And (FragmentNotIn sa orig) (FragmentNotIn sb orig)
| FragmentNotIn (Fragment c :> sa) orig = TypeError (NotUniqueFragmentInApi orig)
| FragmentNotIn (x :> sa) orig = FragmentNotIn sa orig
| FragmentNotIn (Fragment c) orig = TypeError (NotUniqueFragmentInApi orig)
| FragmentNotIn x orig = ()
| :}
>>> :kind! FragmentNotIn (Fragment Int :> Get '[JSON] NoContent) (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent)
FragmentNotIn (Fragment Int :> Get '[JSON] NoContent) (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent) :: Constraint
= (TypeError ...)
>>> :kind! FragmentNotIn (Get '[JSON] NoContent) (Fragment Int :> Get '[JSON] NoContent)
FragmentNotIn (Get '[JSON] NoContent) (Fragment Int :> Get '[JSON] NoContent) :: Constraint
= () :: Constraint
Good!
Now, define type family for checking uniqueness in whole API:
>>> :{
| type family FragmentUnique api :: Constraint where
| FragmentUnique (sa :<|> sb) = And (FragmentUnique sa) (FragmentUnique sb)
| FragmentUnique (Fragment a :> sa) = FragmentNotIn sa (Fragment a :> sa)
| FragmentUnique (x :> sa) = FragmentUnique sa
| FragmentUnique (Fragment a) = ()
| FragmentUnique x = ()
| :}
In order to make this constraint work on type-level I have to add extra type class.
>>> class FragmentUnique api => AtLeastOneFragment api
<interactive>:92:1: error:
• Potential superclass cycle for ‘AtLeastOneFragment’
one of whose superclass constraints is headed by a type family:
‘FragmentUnique api’
Use UndecidableSuperClasses to accept this
• In the class declaration for ‘AtLeastOneFragment’
GHC tells me to enable UndecidableSuperClasses
extension!
>>> :set -XUndecidableSuperClasses
>>> class FragmentUnique api => AtLeastOneFragment api
Good!
Now, test instances for different APIs:
>>> instance AtLeastOneFragment (Fragment Bool :> Get '[JSON] NoContent)
<interactive>:95:10: error:
• Illegal instance declaration for
‘AtLeastOneFragment (Fragment Bool :> Get '[JSON] NoContent)’
(All instance types must be of the form (T a1 ... an)
where a1 ... an are *distinct type variables*,
and each type variable appears at most once in the instance head.
Use FlexibleInstances if you want to disable this.)
• In the instance declaration for
‘AtLeastOneFragment (Fragment Bool :> Get '[JSON] NoContent)’
Again, enable FlexibleInstances
extension.
>>> :set -XFlexibleInstances
>>> instance AtLeastOneFragment (Fragment Bool :> Get '[JSON] NoContent)
>>> instance AtLeastOneFragment (Fragment Bool :> Fragment Int :> Get '[JSON] NoContent)
<interactive>:98:10: error:
• Only one Fragment allowed per endpoint in api ‘Fragment Bool
:> (Fragment Int :> Get '[JSON] NoContent)’.
• In the instance declaration for
‘AtLeastOneFragment
(Fragment Bool :> (Fragment Int :> Get '[JSON] NoContent))’
Now I have type class that raises type errors when several fragments would be in API. And these kind of errors would be catched during compile time. Awesome! (Requirements #4, #5, #6, #7).
Since Fragment is a new API combinator, I have to provide the instance of HasClient
for it (requirement #3).
class RunClient m => HasClient m api where
type Client (m :: * -> *) (api :: *) :: *
clientWithRoute :: Proxy m -> Proxy api -> Request -> Client m api
hoistClientMonad
:: Proxy m
-> Proxy api
-> (forall x. mon x -> mon' x)
-> Client mon api
-> Client mon' api
Client
stands for producing type of a function for querying API.clientWithRoute
populates Request
while traversing API.hoistClientMonad
is the way to hoist client from one monad to another.AtLeastOneFragment
constraint used here to restrict API with fragments and raise an error described in previous section (requirement #3, client-side).
instance ( AtLeastOneFragment api, HasClient m api
, FragmentUnique (Fragment a :> api)
) => HasClient m (Fragment a :> api) where
type Client m (Fragment a :> api) = Client m api
clientWithRoute pm _ = clientWithRoute pm (Proxy :: Proxy api)
hoistClientMonad pm _ = hoistClientMonad pm (Proxy :: Proxy api)
Server API processed in HasServer
type class:
class HasServer api context where
type ServerT api (m :: * -> *) :: *
route ::
Proxy api
-> Context context
-> Delayed env (Server api)
-> Router env
hoistServerWithContext
:: Proxy api
-> Proxy context
-> (forall x. m x -> n x)
-> ServerT api m
-> ServerT api n
ServerT
represents the API type.route
takes API, context, Delayed env (Server api)
and produces the Router. Delayed
is a representation of a handler with scheduled delayed checks that can trigger errors. See its documentation for further explanation. Router
stands for Application
.hoistServerWithContext
is the way to hoist server from one monad to another with context provided by user.instance ( AtLeastOneFragment api, HasServer api context
, FragmentUnique (Fragment a1 :> api)
)
=> HasServer (Fragment a1 :> api) context where
Again, AtLeastOneFragment
constraint used here to restrict API with fragments and raise an error described in previous section (requirement #2, server-side).
type ServerT (Fragment a1 :> api) m = ServerT api m
route _ = route (Proxy :: Proxy api)
hoistServerWithContext _ = hoistServerWithContext (Proxy :: Proxy api)
Servant offers ComprehensiveAPI
to cover the most combinators, doctests
to catch compile-time errors and hspec
for functional tests of different packages and components.
:<|> "fragment" :> Fragment Int :> GET
I added this line in definition of a ComprehensiveAPIWithoutStreamingOrRaw'
type and quickly discovered that Fragment
impacts two different packages I have not used before: servant-docs
and servant-foreign
. I realised that ComprehensiveAPI
is a good way to keep the whole set of packages consistent.
With doctest
you can incorporate your tests right into documentation (haddock).
Example of fragment definition:
-- >>> -- /post#TRACKING
-- >>> type MyApi = "post" :> Fragment Text :> Get '[JSON] Tracking
Example of type-level test:
-- >>> type FailAPI = Fragment Bool :> Fragment Int :> Get '[JSON] NoContent
-- >>> instance AtLeastOneFragment FailAPI
-- ...
-- ...Only one Fragment allowed per endpoint in api...
-- ...
-- ...In the instance declaration for...
All imports could be incorporated into scope of module via $setup
named chunk. See documentation for more details.
servant
, servant-client
and servant-server
packages were covered with hspec
tests.
Servant.Links
generates correct URI for specified fragment.Servant.Client
generates correct client according to specified server.Servant.Server
generates correct server handlers.One small piece of URL and a lot of details. I am happy to achieve this simple goal.
Fragment is available on hackage since servant-0.18.2.
I have website built with servant
framework. And I need to add SEO for it. Here we go.
One day I started with a task where I had a choice:
I am talking about two handlers /robots.txt
and /sitemap.xml
useful in SEO optimisations in a project with servant
as dependency. Unlike yesod
, there are no such extensions in servant
ecosystem.
So I had to add/modify the code by hand every time when I need to test some idea.
Otherwise, to write plugin for servant
.
Community opinion about servant
is controversial (based on what I have heard). From user point of view it is simple. And the separation of API and handlers is great! Docs and tutorials are awesome! From development side, servant
considered as type-level fancy stuff. That’s what I heard.
There were no hard issues with using servant
. So I become curious. Is it really true about development side? The only way to find is to resolve my task this way and to share thoughts and findings with you.
I had only one hour per day (evenings, mostly). So I started.
Robots.txt is plaintext file that could be statically or dynamically served from web server via /robots.txt
handler. It gives instructions to robots what should be indexed and what should not.
User-agent: *
Disallow: /static
Sitemap: https://example.com/sitemap.xml
Here is an example of content I want to receive from API. It means that robots with any user-agent should not be allowed to index endpoints starting with /static
. And sitemap is located by the following URL.
Robots specification is not limited by these commands. But for the start it would be enough. As a user, I want to specify Disallow
as a keyword somewhere in API.
Sitemap.xml is an XML file that could contain either list of URLs of pages that should be indexed by robots or list of nested sitemaps URLs if website index is large enough.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com</loc>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
There is a /urlset/url
list with an URL inside and some optional parameters. I want to use Frequency
(i.e. /urlset/url/changefreq
) and Priority
(/urlset/url/priority
) as keywords in API as well. And XML should be rendered somehow from API.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://www.example.com/sitemap/1/sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>http://www.example.com/sitemap/2/sitemap.xml</loc>
</sitemap>
</sitemapindex>
There is a /sitemapindex/sitemap
list with sitemap index. Each URL (//loc
) from this index represents a single part of whole sitemap. If sitemap is large enough then sitemap index should be built instead. Each index location should contain no more than fifty thousands target locations (HTML page URLs). Sitemap specification contains more details.
In order to start the implementation I asked myself how final API should look like? What it should be like? And I came with something like:
-- ** Example API
type PublicAPI
= Get '[HTML] HomePage
:<|> ("blog" :> Frequency 'Always :> BlogAPI)
:<|> ("news" :> Capture ":newsurl" NewsUrl :> Get '[HTML] NewsPage)
:<|> ("search" :> QueryParam "q" SearchPattern :> Get '[HTML] SearchResultsPage)
:<|> ("about" :> Priority '(1,0) :> Get '[HTML] AboutPage)
:<|> "auth" :> ReqBody '[JSON] Login :> Post '[JSON] NoContent
type BlogAPI
= Capture ":blogurl" BlogUrl :> Get '[HTML] BlogPage
:<|> Capture ":blogurl" BlogUrl
:> ReqBody '[JSON] BlogComment
:> Post '[JSON, HTML] BlogPage
type ProtectedAPI = Disallow "admin" :> Get '[HTML] AdminPage
type StaticAPI = "cdn" :> Disallow "static" :> Raw
type API = StaticAPI :<|> ProtectedAPI :<|> PublicAPI
Looks like I need to encode Disallow
, Frequency
and Priority
into API without any impact on serving.
-- ** Example app
startServer :: IO ()
startServer = do
Warp.runSettings Warp.defaultSettings
$ serveWithSeo website api server
where
website = "http://example.com"
And I should find the way to derive both robots.txt
and sitemap.xml
with as less code as possible from user perspective. E.g. only one function serveWithSeo
should be called.
Next step was to ask: what really should be implemented and how to achieve it? The only way to become comfortable with the task is to decompose it to degree where the each subtask from the tree would become transparent for you.
I group all requirements on several components:
Disallow
, Frequency
and Priority
are the starting point in my journey.Robots
data type. There should be the way to transform the final API into intermediate Robots
representation.Sitemap
data type. The same as previous one but for Sitemap
.Robots
and Sitemap
. Handlers with default renderers should be provided from intermediate data types to target content types.I was starting to notice how organically types drive me throughout the gathering of functional requirements.
Disallow
should wrap the path piece of URL (i.e. symbol).Disallow
should not affect servant-server
functionality.Disallow
should be gathered to Robots
intermediate data type for each API branch.Disallow
should invalidate Sitemap
for whole API branch if it is present in API branch.Frequency
should be used with :>
combinator.Frequency
should have (on type level) one parameter with following available values (according to sitemap spec): never
, yearly
, monthly
, weekly
, daily
, hourly
, always
.Frequency
should not affect servant-server
functionality.Frequency
should be gathered for each URL in contained API branch.Frequency
values in one API branch, the outer-most should be used, i.e. overwrite rule.Priority
should be used with :>
combinator.Priority
should have (on type level) one parameter with a value representing priority according to sitemap spec, i.e. in range between 0.1 and 1.0.Priority
should not affect servant-server
functionality.Priority
should be gathered for each URL in contained API branch.Priority
values in one API branch, the outer-most should be used, i.e. overwrite rule.Robots
data typeI called it RobotsInfo
.
RobotsInfo
should contain information about every API branch where Disallow
appeared, i.e. list of disallowed path pieces.RobotsInfo
should contain knowledge about sitemap presence in API (present or not).Sitemap
data type.I called it SitemapInfo
.
SitemapInfo
should contain list of sitemap entries (i.e. [SitemapEntry]
).SitemapEntry
should represent particular API branch.SitemapEntry
should contain information about all pieces from which list of URLs could be constructed.SitemapEntry
should contain information about all query parameters from which list of URLs could be constructed.SitemapEntry
might contain Priority
or not (optional parameter).SitemapEntry
might contain Frequency
or not (optional parameter).SitemapInfo
should be automatically gathered once there is Get '[HTML] a
in API present.SitemapInfo
should be created for such API branches.userType
from Capture' mods sym userType
(we will discuss it later).userType
from QueryParam' mods sym userType
(we will discuss it later).SitemapInfo
could be splitted on sitemap index where every key from index should have reproducible set of corresponding URLs when its length is more than 50000.RobotsInfo
.SitemapInfo
./robots.txt
./sitemap.xml
./sitemap/:sitemapIndex/sitemap.xml
.RobotsInfo
to its textual representation).HTTP 404 NOT FOUND
while querying /sitemap/:sitemapIndex/sitemap.xml
).I decided to choose at least one key per API branch. And if one particular branch will have more than 50000 URLs then split it on several keys/parts.
There are several levels of context. For simplicity I draw them on this diagram.
There was one strange thing across the design. I started to see the pattern. How to implement these requirements without previous type-level experience. But I was not sure. I stepped back and read the brilliant book Thinking with Types
by Sandy Maguire. I looked in servant-server
, servant-swagger
sources and Restricting servant query parameters by Alexander Vershilov as guidelines. These insights showed me how to deal with the task.
RobotsInfo
and SitemapInfo
should both be Monoid
and Semigroup
to be retrieved and combined altogether in their final representation.Proxy
is the way to retrieve information from type level to data level.Capture
and QueryParam
could be handled with a separate type classes (25 and 26 requirements). User must implement instances for corresponding type classes for a function that looks like MonadIO m => Proxy a -> app -> m [a]
. Lists of possible values will be further propagated in Capture'
and QueryParam'
instances.PolyKinds
. When I needed types of different kinds GHC does not tell me anything about it.I have heard a lot about Simple Haskell, Boring Haskell, Fancy Haskell. I realized that restrictions and limitations produced for good reasons: TTM, benefits vs. costs, risks and so on.
On the contrary, there are some awesome explanations from problem solving point of view. They lead to specific technologies or even “type-level magic” stuff.
These are different mindsets. They all share that problem have to be solved. Difference is in the way to solve the problems. Maybe different problems. Maybe higher-order problems. Maybe single task from long awaited todo-list.
One day you can find yourself in a situation when you have to deliver faster than usual. Another day there are no hard time boundaries and you feel relaxed. We are living in the endless uncertainty of estimations and expectations. We are living in chaos where nothing could be predicted.
Sometimes we know what should be done and we have to do the things we did before. Sometimes task feels like extremely new: new domain, interfaces, requirements, specs. Abstractions are quite good way to dealing with them. It is a single side of coin.
When you add team to consideration, situation could become even worse. In large projects it is absolutely normal that you spend on documenting/analysing/coding only 10% of business hours. Rest of time are about communications, mostly. Communications with business partners, with BA folks, with development team, with management, with QA team, with executives, with users.
I feel myself like I am simultaneously living in a few worlds. And it is hard to express the concerns to somebody because I have to understand what are the preconditions, what context is suitable for people I interact with, what goals they are pursuing, what agenda is on their tables. And it is impossible. In the village where I grew up, neighbours used to say: “Going to politics? - Good riddance!”
And I am starting to feel myself unhappy from the beginning of interaction. So the best solution for me in these situations is to remain silent. Unless, I am experiencing mentioned trade-offs by myself.
There is a force bigger than us. Nothing could stop it. We cannot resist it. We could only accept it and let it drive us all the way.
Since the beginning I struggled a lot, there were:
Despite all of that, the goal was achieved. It finally lets me go.
Results are available as a library:
I encourage you to build useful website with servant and hundred of thousands pages and to be successfully indexed by robots!
If you have the notes to add or you see some issues in the spec/code, please contact me, e.g. on Github.
Thank you and have a good day!
servant-server
.servant-swagger
.This year started fenomenally: I became the fay
compiler maintainer.
I have a small side project built with ghc
, yesod
, stack
and fay
. And yesod-fay
. One day after migration to GHC 8 I corrupted .stack-work
and decided to recreate it. After that errors appeared from fay
side.
Could not find module
or even
ghc-pkg: Could not find package
Or something like that.
In fay
wiki stated that it is necessary to set up HASKELL_PACKAGE_SANDBOX
variable. The point is that stack
usually takes care of it and rewrite your value with a new one.
stack path
Command above will give you a clue. It happened because there are several package databases.
~/.cabal
.~/.stack
.~/path/to/your/project/.stack-work
.Before GHC 8 the main idea of how to avoid cabal-hell was cabal sandbox
. And it was reflected in fay
wiki. When stack
appeared, it takes management of local dependencies and even package database by itself.
Thus, during fay compilation stack
decided to switch from local package to its global. And lookup for dependent package was failed.
Since fay
does not have typechecker, it relies on GHC and uses ghc-paths
. Under the hood there was usage of ghc-pkg
and stack
package db. There was a mess.
I decided to implement checks for ghc-pkg
usage on both sides: fay
and yesod-fay
. Manual compilation should work in stack project. Compilation through the Fay API should work as well.
Finally I was able to fix two packages. And become maintainter of them and move on.
With GHC 8.4 and further there is another approach to build packages: cabal new-build
or cabal v2-build
. And it seems that ghc-pkg
should be adopted again.
Update 2020-06-16: Thanks to Jordan Okene, this post and associated github repo were updated to LTS-16 and webdriver-0.9.
As a software engineer I have heard a lot about test automation and even about Selenium. But never tried it. Since I have an outstanding activity with web scrapping I tried instead of emulating web browsers to launch them in the automatic way. And Selenium should be the best choice for it.
What is it? You have the web browser
that supports Selenium
via webdriver
implementation as interface for Selenium
. You have a Selenium server
. And you have your client test automation app
with necessary scenarios inside.
How it works?
client test automation app
sends requests that are complaint with Selenium server
to it.Selenium server
handles these requests and invoke web browser
through webdriver
and receives response from browser
.Selenium server
transfers response to your client app
as a proxy middleware.And that’s actually it. With Haskell’s REPL ghci
I can show you how to build test automation scenarios interactively and ship them into single test case. Issues/questions raised on webdriver
package github page motivated me to write this blogpost.
stack
, selenium standalone server
, chromedriver
.
stack
is a tool that will help you to avoid all possible complexity with Haskell initial setup if you are not familiar with Haskell. You can obtain it from its download page.
selenium standalone server
could be downloaded from official site.
chromedriver
could be obtained from here.
Once you followed all instructions, set up ${JAVA_HOME}
(or %JAVA_HOME%
in Windows) environmental variable, put chromedriver
and stack
executables somewhere visible, i.e. in $PATH
, you can enjoy playing with it. We will use customized webdriver
haskell package (see Cookies
section on this page for reasons of using custom code instead of published).
$ stack new selenium-example
Modify selenium-example/stack.yaml
configuration file by modifying extra-deps
section:
extra-deps:
- webdriver-0.9.0.1
If in your project directory you do have package.yaml
file then add a dependency right there:
dependencies:
- base >= 4.7 && < 5
- webdriver <= 0.9.0.1
- text <= 1.2.3.0
Otherwise, modify selenium-example.cabal
file library/build-depends
subsection:
build-depends: base >=4.7 && <5
, webdriver <= 0.9.0.1
, text <= 1.2.3.0
Launch following command and wait a couple minutes to finish.
$ stack build
By default you have a couple of files in your project dir. What you need is to replace Main module app/Main.hs
with following content:
module Main where
import Lib
import Data.Text
import Test.WebDriver
import Test.WebDriver.Session
import Test.WebDriver.Commands
import Test.WebDriver.JSON (ignoreReturn)
chromeConfig :: WDConfig
chromeConfig = useBrowser chr defaultConfig
{ wdHost = "0.0.0.0", wdPort = 4444, wdHTTPRetryCount = 50 }
where chr = chrome
main :: IO ()
main = someFunc
Now we are ready to go!
Step 1. Start selenium standalone server
in console by running: java -jar ~/Downloads/selenium-server-standalone-x.x.x.jar
where x.x.x
is a version of downloaded selenium server. You can verify it by opening http://localhost:4444/ in your browser.
Step 2. Launch your client from console:
$ stack repl
The following GHC options are incompatible with GHCi...
Configuring GHCi with the following packages: selenium-example
Using main module: 1. Package `selenium-example' component ...
GHCi, version 8.4.3: http://www.haskell.org/ghc/ :? for help
[1 of 2] Compiling Lib ( .../src/Lib.hs, interpreted )
[2 of 2] Compiling Main ( .../app/Main.hs, interpreted )
Ok, two modules loaded.
Loaded GHCi configuration from ...
*Main Lib>
Ok, we are ready to go further.
Session in terms of Selenium is the test scenario with particular browser instance. Each session triggers new browser. By default all sessions are available in your browser by the http://localhost:4444.
Step 3. Launch your web browser interactively in your client opened in Step 1:
*Main Lib> session <- runSession chromeConfig $ getSession
*Main Lib>
We started new session with configured chrome browser. Now we are going to interact with web browser by session
that we received by getSession
.
*Main Lib> runWD session (openPage "http://localhost:4444")
*Main Lib>
We passed our session and webdriver action in brackets (opening selenium start page) into webdriver context using runWD
.
*Main Lib> runWD session (openPage "https://google.com")
*Main Lib> :t findElem
findElem
:: Test.WebDriver.Class.WebDriver wd => Selector -> wd Element
*Main Lib> :i Selector
data Selector
= ById text-1.2.3.0:Data.Text.Internal.Text
| ByName text-1.2.3.0:Data.Text.Internal.Text
| ByClass text-1.2.3.0:Data.Text.Internal.Text
| ByTag text-1.2.3.0:Data.Text.Internal.Text
| ByLinkText text-1.2.3.0:Data.Text.Internal.Text
| ByPartialLinkText text-1.2.3.0:Data.Text.Internal.Text
| ByCSS text-1.2.3.0:Data.Text.Internal.Text
| ByXPath text-1.2.3.0:Data.Text.Internal.Text
-- Defined in ‘Test.WebDriver.Commands’
instance Eq Selector -- Defined in ‘Test.WebDriver.Commands’
instance Ord Selector -- Defined in ‘Test.WebDriver.Commands’
instance Show Selector -- Defined in ‘Test.WebDriver.Commands’
*Main Lib> searchInput <- runWD session (findElem (ById "lst-ib"))
<interactive>:6:46: error:
• Couldn't match expected type ‘text-1.2.3.0:Data.Text.Internal.Text’
with actual type ‘[Char]’
• In the first argument of ‘ById’, namely ‘"lst-ib"’
In the first argument of ‘findElem’, namely ‘(ById "lst-ib")’
In the second argument of ‘runWD’, namely
‘(findElem (ById "lst-ib"))’
*Main Lib> :set -XOverloadedStrings
*Main Lib> searchInput <- runWD session (findElem (ById "lst-ib"))
*Main Lib>
Sending text to input
is very easy with sendKeys
. Check the type of sendKeys
and send something to google.
*Main Lib> :t sendKeys
sendKeys
:: Test.WebDriver.Class.WebDriver wd =>
text-1.2.3.0:Data.Text.Internal.Text -> Element -> wd ()
*Main Lib> runWD session (sendKeys "Glasgow\n" searchInput)
Let’s find Wikipedia on result page.
*Main Lib> :t findElems
findElems
:: Test.WebDriver.Class.WebDriver wd => Selector -> wd [Element]
*Main Lib> links <- runWD session (findElems (ByClass "r"))
*Main Lib> :t links
links :: [Element]
*Main Lib> :t getText
getText
:: Test.WebDriver.Class.WebDriver wd =>
Element -> wd text-1.2.3.0:Data.Text.Internal.Text
*Main Lib> linkTexts <- runWD session (mapM getText links)
*Main Lib> :t linkTexts
linkTexts :: [text-1.2.3.0:Data.Text.Internal.Text]
*Main Lib> print linkTexts
["Glasgow - Wikipedia","People Make Glasgow | Official Guide to the
City of Glasgow","First Time Must Sees in Glasgow | People Make
Glasgow","University of Glasgow - homepage","Glasgow 2018: Best of
Glasgow, Scotland Tourism - TripAdvisor","Things to do in
Glasgow","Glasgow travel - Lonely Planet","Glasgow City
Council","Glasgow - Holidays, City & Weekend Breaks | VisitScotland"]
*Main Lib>
*Main Lib> runWD session (ignoreReturn $ executeJS [] "alert('custom alert');")
*Main Lib>
*Main Lib> myCookies <- runWD session (cookies)
*Main Lib> :t myCookies
myCookies :: [Cookie]
*Main Lib> length myCookies
5
*Main Lib> print myCookies
[Cookie {cookName = "1P_JAR", cookValue = "2018-8-3-10", cookPath =
Just "/", cookDomain = Just ".google.com", cookSecure = Just False,
cookExpiry = Just 1.535885639e9},Cookie {cookName = "UULE", cookValue
=
"a+cm9sZToxIHByb2R1Y2VyOjEyIHByb3ZlbmFuY2U6NiB0aW1lc3RhbXA6MTUzMzI==",
cookPath = Just "/", cookDomain = Just "www.google.com", cookSecure =
Just False, cookExpiry = Just 1.533380041e9},Cookie {cookName = "NID",
cookValue =
"135=e6Jaak3KFazKszUmf60YNU45V_rp_Zbzw3t6zSvHWNzvuDHmN5d_Vf9iNGfpT",
cookPath = Just "/", cookDomain = Just ".google.com", cookSecure =
Just False, cookExpiry = Just 1.548845639106112e9},Cookie {cookName =
"DV", cookValue = "4xCQLaMEo_YeoPugrFVU4MaNxMP2TxY", cookPath = Just
"/", cookDomain = Just "www.google.com", cookSecure = Just False,
cookExpiry = Just 1.533294239e9},Cookie {cookName = "CGIC", cookValue
=
"IlV0ZXh0L2h0bWwsYXBwbGljYXRpb24veGh0bWwreG1sLGFwcGxpY2F0aW9uL3htb",
cookPath = Just "/search", cookDomain = Just ".google.com", cookSecure
= Just False, cookExpiry = Just 1.549072114233595e9}]
*Main Lib>
Let’s review expiration date of cookie. It’s Just
a unix timestamp.
*Main Lib> runWD session closeSession
Interactive mode provided by stack repl
is better choice to create your automation tasks on flight. But how to combine them altogether to execute later? In ghci
you have a history so you need to move it somehow and combine in single task. Moreover, you can avoid multiple repeatition of runWD session ...
by using do-notation since WD ()
is a monad.
Replace following code with main function
in your test application:
main :: IO ()
main = do
result <- runSession chromeConfig googleIt
print result
googleIt :: WD [Text]
googleIt = do
openPage "https://google.com"
searchInput <- findElem (ById "lst-ib")
sendKeys "Glasgow\n" searchInput
links <- findElems (ByClass "r")
linkTexts <- mapM getText links
closeSession
return linkTexts
We hide session inside googleIt
function that should return list of results. Reload our repl
and let’s try again.
*Main Lib> :r
Ok, two modules loaded.
*Main Lib> main
["Glasgow - Wikipedia","People Make Glasgow | Official Guide to the
City of Glasgow","First Time Must Sees in Glasgow | People Make
Glasgow","University of Glasgow - homepage","Glasgow 2018: Best of
Glasgow, Scotland Tourism - TripAdvisor","Things to do in
Glasgow","Glasgow travel - Lonely Planet","Glasgow City
Council","Glasgow - Holidays, City & Weekend Breaks | VisitScotland"]
You can actually run chrome in headless mode by simple adding appropriate flags or even switching to chromium:
chromiumConfig :: WDConfig
chromiumConfig =
useBrowser chr defaultConfig
{ wdHost = "0.0.0.0", wdPort = 4444, wdHTTPRetryCount = 50 }
where chr = chrome
{ chromeBinary = Just "/usr/bin/chromium"
, chromeOptions = ["--headless"
, "--mute-audio"
, "--disable-gpu"
, "--no-sandbox"
]
}
Using selenium-standalone-3.7.1.jar you can try to launch PhantomJS browser:
phantomConfig = useBrowser phantom defaultConfig
where phantom =
Phantomjs
{ phantomjsBinary = Just "/opt/phantomjs/bin/phantomjs"
, phantomjsOptions =
[ "/opt/phantomjs/src/ghostdriver/main.js"
, "--webdriver=8910"
, "--webdriver-selenium-grid-hub=http://127.0.0.1:4444"
, "--webdriver-logfile=/var/log/phantomjs/webdriver.log"
]
}
And one more headless browser:
htmlUnitConfig = useBrowser HTMLUnit defaultConfig
Actually PhantomJS
is currently abandoned. And HtmlUnit
cannot interact with React.js
. However, maybe you will be lucky.
Example is fully available on my github.
In case of questions or comments please contact me on Twitter: @marunarh.
Please aware that this blog was previously written in Russian. I will not adopt previous articles but try to use blog to describe state of my mind. I really love cool success stories about software and find interesting the whole process: from generation of some awesome idea through the thrilling details of its implementation till its delivery and going with a live system.
In this blog post you can find the story about small project which purpose is to make calculations about visa restrictions and limitations (if it’s applicable for your country when you are going to travel abroad). Starting from the beginning only time I had was 10 minutes per day or, actually, per week. Once sceleton and algorithm were implemented in some way I started to be excited of final result. And here we are. If you want to try it right now please use following link: https://an-pro.org/tools/visa-calculator
Consider you are frequent traveller. You have Shengen visa (or any visa) which allows you to stay somewhere X days during shift for Y days, e.g. 30 days per 90. It means that when you reach first limit you have to wait from home extra 60 days (90 days in total) to decrease first limit (30) and get new free days to plan your return.
For Shengen visa there is calculator. I found it quite tricky, working in unstable way and making all calculations on server side. I thought there is no need to make any requests to server. They are redundant by definition.
Another case appeared when I was in a business trip in Ecuador. We were discussing with teammates how often we are travelling abroad and how many days left to become a non-resident. I don’t pretty know how it’s going in your country but I can say about Russia. Non-resident status will increase your taxes as a citizen from 13% to 30% till you get out from this period. Ah, limit is half a year per year.
The idea is simple: do not break the visa rules and try not to become non-resident. You have to do a lot of excel calculations to count all that days in a different ways:
I had some Haskell experience to try it in a browser. Somehow, I discovered few alternatives like Elm and PureScript. I followed advise to try PureScript when it was quite unstable, one year ago there was compiler upgrade and tooling transition through dependency hell (0.10 to 0.11 if my memories are correct). Ok. What’s next?
Frameworks: Pux vs. Halogen? Halogen did not worked for me (I hope because of major upgrade). But Pux worked fine and I focused on it. If you are familiar with MVC then rest will be obvious.
So I started from top to bottom. And when I reached the algorithm situation became tough. There was user input. There were some pretty well validations: date format, date range compliance and number of days format and compliance. User could give array of ranges and two numbers. Good. What’s next. I decided there will be no explicit button. Every input should trigger validations and calcluations if it’s possible.
Once data reached validations what should be the result? How can I represent it? What it actually should be? After some analysis I realized that visa could have several types from user input perspective, that result could be both positive and negative. Which date from future range (which one?) should be taken as a start point?
I did not understand pretty well what I am going to produce. After answering all that questions following State was discovered:
newtype State = State
{ title :: String
, route :: Route
, loaded :: Boolean
, userRanges :: Array UserRange
, today :: MaybeDate
, visa :: Visa
, ranges :: Array DateRange
, nextId :: Int
, result :: String
}
newtype UserRange = UserRange
{ start :: DateWidget
, end :: DateWidget
, id :: Int
, msg :: ErrorMsg
}
newtype DateRange = DateRange
{ start :: MaybeDate
, end :: MaybeDate
, id :: Int
, diff :: MaybeInt
}
newtype DateWidget = DateWidget
{ widgetDate :: String, widgetMsg :: ErrorMsg }
MaybeDate
, MaybeInt
and ErrorMsg
are the simple newtype wrappers to avoid orphans.
User might add and/or remove previous, planned trips and visa restrictions. When data passed all validations Calculation started immediately.
Client application received visa parameters, user ranges and validated date ranges. Client started with following steps:
https://an-pro.org/tools/visa-calculator
I have plans to implement following features:
I will really appreciate your feedback.
Thank you.
С ещё большим удовольствием посетил FProg November Meetup.
Узнал знакомых по предыдущему митапу людей. Впечатления остались скорее смешанные, как и в прошлый раз. И далее следует как раз отделить весь сумбур от достойных докладов так же, как побочные эффекты отделяют от чистоты.
Всё-таки, тут те же проблемы, что и за рубежом, т.к. люди одни и те же. Только вскрываются они на порядок позднее. В USA python-разработчик женского пола уже успела публично обвинить в сексизме докладчика, того уже успели выпереть из конторы, её после таких заявлений понесло, а впоследствии и она лишилась работы. Вот до чего доводят эффекты на публику.
На ноябрьском докладе можно было слышать как шуточки сексистского толка, призванные подчеркнуть меметичность стереотипов, так и раздражённую реакцию на них. Личная позиция автора статьи индифферентна как к таким шуткам, так и к реакции до тех пор, пока не начались публичные обвинения. Как только воспроизведутся публичные обвинения, позиция автора изменится до резко негативной по отношению к обвиняющей стороне. Причины подобных возможных изменений должны быть очевидны.
Ведь вполне возможно, что и сам я выходил или выйду в будущем за рамки нейтралитета. В таком случае сам же и виноват. Теперь к докладу.
Ссылка: не найдена; докладчик: @rigidus
@Rigidus уверенно задал хороший годный темп тёплым ноябрьским вечером и поднял планку, так что сразу стало ясно, что вечер удастся.
Доклад был расчитан на людей, не знакомых с Lisp. Каждая скользкая тема в аспекте Common Lisp была разъяснена в той или иной мере. Не вполне понял про гигиеничность макросов, тут собственная неграмотность подкачала.
Макросы в Lisp - едва ли не самые мощные макросы среди всех ЯП.
Ссылка: https://github.com/mariyadavydova/notes/blob/master/th/th.pdf; докладчик: Мария Давыдова, JetBrains
До этого как-то сам собою использовал Template Haskell, подключая код из внешних файлов с соответствующим расширением. Но не задумывался, что под капотом.
А хорошо поставленный доклад Марии расставил все точки над i. Посмотрели на деревья, на код, немного коснулись парсеров. Осталось лишь попробовать что-нибудь качественно распарсить. Может J?!
Ссылка: https://github.com/si14/fprog-2015-11; докладчик: Дмитрий Грошев
Доклад не вписывался в общую тематику, но был на не менее интересную тему: солверы
. А точнее на высокоуровневый MiniZinc, который в недрах своих дёргает API целого множества разношёрстных и разномастных солверов для выбора наиболее оптимального наиболее шустрого решения задачи, условия которой задаются на лету.
“Пойди туда - не знаю, куда; принеси то - не знаю, что!” - вот про решение подобных задач и был доклад. Очень хороший темп, live coding - всё это очень удачно разнообразило программу конференции.
Ссылка: https://github.com/machinezone/mzbench; докладчик: Ренат Идрисов, Machine Zone
Из Новосибирска к нам приехал Ренат и рассказал про макросы в Эрланге, про диалект Lisp, написанный на Erlang, про переопределение поведения интерпретатора и прочие интересные штуки.
К сожалению, мозг потребовал дозарядку, презентация была тускловатой, а свет был слишком мощным, поэтому глаза пришлось вынимать и промывать отдельно.
Ссылка: https://www.youtube.com/watch?v=xd2xtQ61zKo; докладчик: Михаил Муцянко, JetBrains
Во время доклада Михаил упомянал о выступлении на FP Conf. На своей территории он говорил о том же. В принципе не нужно слов, по ссылке всё сами увидите.
Доклад потрясающий.
Во время разработки front-end вообще и использования фреймворка Yesod в частности приходится иметь дело с JavaScript. Разумеется, пока. Дело в том, что для работы с JS рекомендуется использовать Shakespearean Template. А значит, куски JS будут лежать либо в файлах с расширением .julius, либо в Template Haskell. Проблема удобства разработки встаёт тут же. Yesod предоставляет живую перезагрузку кода. Тем самым, цикл разработки становится следующим:
А значит, если у нас в коде ошибка, то мы отловим её в runtime. Вдобавок, сказывается отсутствие REPL для JS. Если поднять cabal repl
, то из ghci нереально будет проверить JS-код.
Что же делать? Как сместить баги из runtime в compile-time? Как добавить в ужас JS немного статической типизации?
На эти вопросы уже есть ответы.
GHCJS, Elm - судя по отзывам, превосходные вещи. Однако мне не довелось ещё соприкоснуться с ними. У Сноймана были какие-то мысли на их счёт, но я не вдавался в подробности и не могу ничего сказать на сей счёт конкретного.
Fay - это подмножество Haskell, которое можно успешно скомпилировать в валидный JS код. Среди возможностей Fay отмечают:
Как можно заметить из их wiki, он успешно интегрирован во многие вэб-фреймворки. Но остановимся на Yesod.
Создадим обычный проект на yesod:
$ yesod init
Welcome to the Yesod scaffolder.
I'm going to be creating a skeleton Yesod project for you.
What do you want to call your project? We'll use this for the cabal name.
Project name: faytest
Yesod uses Persistent for its (you guessed it) persistence layer.
This tool will build in either SQLite or PostgreSQL or MongoDB support for you.
We recommend starting with SQLite: it has no dependencies.
s = sqlite
p = postgresql
pf = postgresql + Fay (experimental)
mongo = mongodb
mysql = MySQL
simple = no database, no auth
mini = bare bones, the "Hello World" of multi-file Yesod apps
(Note: not configured to work with yesod devel)
url = Let me specify URL containing a site (advanced)
So, what'll it be? pf
That's it! I'm creating your files now...
---------------------------------------
___
{-) |\
[m,].-"-. /
[][__][__] \(/\__/\)/
[__][__][__][__]~~~~ | |
[][__][__][__][__][] / |
[__][__][__][__][__]| /| |
[][__][__][__][__][]| || | ~~~~
ejm [__][__][__][__][__]__,__, \__/
---------------------------------------
The foundation for your web application has been built.
There are a lot of resources to help you use Yesod.
Start with the book: http://www.yesodweb.com/book
Take part in the community: http://yesodweb.com/page/community
It's highly recommended to follow the quick start guide for
installing Yesod: http://www.yesodweb.com/page/quickstart
If your system is already configured correctly, please run:
cd faytest && stack build && stack exec -- yesod devel
Выберем pf
…
Тут следует отметить, что для работы с PostgreSQL, необходимо, чтобы были разрешены следующие зависимости:
postgres
в СУБД, если вы этого ещё не сделали ранее.После того, как зависимости разрешены, создаём базу данных и юзера.
Подсоединяемся к СУБД.
Создаём базу и юзера.
postgres=# create database faytest;
CREATE DATABASE
postgres=# create user faytest with password 'faytest';
CREATE ROLE
Теперь заводим проект. Как поднимется, смотрим тут.
Последним пунктом значится вычисление числа Фибоначчи по его индексу. А теперь посмотрим, как это реализовано.
В структуре проекта появились следующие файлы:
./static/faygen-omEJ2n4J.js
./static/fay-runtime.js
./fay-shared
./fay-shared/SharedTypes.hs
./fay
./fay/FFIExample.hs
./fay/Home.hs
./fay/Fay
./fay/Fay/Yesod.hs
В шаблоне templates/homepage.hamlet
находим строки:
Текстовое поле для ввода и вывода значений. Где же описана логика?
В обработчике Handler/Home.hs
находим строку $(fayFile "Home")
. Так подключается fay/Home.hs
.
Посмотрим на содержимое:
{-# LANGUAGE RebindableSyntax #-}
{-# LANGUAGE OverloadedStrings #-}
module Home where
import FFIExample
import DOM
import Data.Text (fromString)
import qualified Data.Text as T
import Fay.Yesod
import Prelude
import SharedTypes
main :: Fay ()
main = do
input <- getElementById "fibindex"
result <- getElementById "fibresult"
onKeyUp input $ do
indexS <- getValue input
index <- parseInt indexS
call (GetFib index) $ setInnerHTML result . T.pack . show
С помощью монады Fay осуществляются все преобразования с побочными эффектами, прямо как с IO. Не правда ли, код стал более читаем? Часть функций мы импортировали из DOM
, call
- из Fay.Yesod
, остальное - из FFIExample
. Как можно заметить, у нас тут и клиентская, и серверная часть. Как водится, они разделены. С помощью call
мы делаем асинхронную отправку запроса по адресу /fay-command
. Сервер принимает запрос и вызывает обработчик appFayCommandHandler
. В Application.hs
прописывается функция onCommand
из Handler.Fay
, которая отрисовывает на клиенте число Фибоначчи по индексу.
Для того, чтобы разобраться с тем, как в Fay реализована клиентская и серверная часть, рассмотрим их по отдельности.
Все файлы, относящиеся к клиентской части лежат в директории fay
:
В Home.hs лежит подключаемый fayFile
. Его содержимое мы уже рассмотрели выше.
В Fay/Yesod.hs
лежат функции для взаимодействия с сервером. Ровно то же самое можно обнаружить в хакадже.
В FFIExample - самое интересное: примеры реализации внешнего интерфейса вызова JS. С побочными действиями, со всеми делами.
-- | Example of defining FFI functions.
--
-- The `ffi' method is currently incompatible with 'RebindableSyntax',
-- so these are defined in another module.
module FFIExample where
import Data.Text (Text)
import DOM
import FFI
onKeyUp :: Element -> Fay () -> Fay ()
onKeyUp = ffi "%1.onkeyup=%2"
setInnerHTML :: Element -> Text -> Fay ()
setInnerHTML = ffi "%1.innerHTML=%2"
Как можно заметить, мы не передаём аргументы в функцию, а используем порядковые номера %1
, %2
и так далее, как в Си.
Все файлы, относящиеся к серверной части, лежат в fay-shared
. В нашем случае это SharedTypes.hs:
{-# LANGUAGE NoImplicitPrelude #-}
{-# LANGUAGE DeriveDataTypeable #-}
module SharedTypes where
import Prelude
import Data.Data
import Fay.Yesod
data Command = GetFib Int (Returns Int)
deriving (Typeable, Data)
Command
здесь - это тип, который принимает сервер и обрабатывает в Handler.Fay
(напомню, что обработчик можно найти по адресу Handler/Fay.hs
). Выше уже это упомяналось, рассмотрим же теперь подробнее:
module Handler.Fay where
import Fay.Convert (readFromFay)
import Import
import Prelude ((!!))
import Yesod.Fay
fibs :: [Int]
fibs = 0 : 1 : zipWith (+) fibs (drop 1 fibs)
onCommand :: CommandHandler App
onCommand render command =
case readFromFay command of
Just (GetFib index r) -> render r $ fibs !! index
Nothing -> invalidArgs ["Invalid command"]
Нетрудно заметить, что при добавлении новых конструкторов типов, можно будет сопоставлением с образцом легко дополнить взаимодействие с сервером так, как того требует бизнес-логика приложения. Да, это устаревающий AJAX, но он ещё не совсем умер. С этим действительно можно работать.
А как это чудо выглядит после компиляции? Помните, в начале в списке файлов проекта появились static/fay-runtime.js
, static/faygen-*.js
? Вот это как раз оно и есть. В файле fay-runtime.js
содержится Prelude
и множество полезных вещей, которые мы можем использовать. В сгенерированном файле содержатся результаты компиляции файлов из директории fay
.
С большим удовольствием посетил FProg Meetup. Посмотрел на людей, проникся атмосферой. Впечатления остались смешанные. Но обо всём по порядку.
В офисе JetBrains мне уже доводилось бывать ранее.. На конференции Saint Perl, 21 декабря 2013 года. Впечатления от площадки и конференции остались самые приятные, хотя и отношение к Perl имел самое посредственное. В этот раз всё было иначе.
Так выглядит типичная работа успешного аналитика. Ему не нужна рабочая станция. Достаточно поставить стол возле окна повыше, на стол поставить пепельницу и положить перед ним пачку сигарет. Можно начинать творить дизайн решения, можно даже надиктовывать его писарям, так как рабочая станция не нужна, а рисовать самому уже не с руки.
Примерно то же самое увидел я, подходя к офису JetBrains. На улице за столиком спиной к окну (деталь несущественная) сидел сотрудник этого замечательного офиса, докуривал сигарету. По выражению лица видно, что solution рождается в муках, решение не даётся просто. А далее произошло то, что на некоторое время выбило меня из колеи и требовало отдельного примирения с реальностью. К такому повороту я не был готов.
Да-да, сотрудник JB, не смущаясь, зажевал потушенную сигарету собственными соплями.
Ребята из JB, если вы читаете это, то знайте, что вы пишете добротный софт, но увиденное раз у вас развидеть обратно уже не получится никогда.
После регистрации мы поднялись наверх в уже знакомую комнату.
Организация была на высшем уровне. Конференция началась уверенно и бодро.
Ссылка: не найдена
Уверенное начало, нормальные слайды, хорошо поставленные примеры. Однако спутанность и прыжки с место на место ближе к середине доклада сбили всю его суть. Не уверен, поняли ли неофиты из доклада основной посыл: простого написания кода недостаточно, нужно ещё думать, что пишешь.
Ссылка: http://leebdeveloper.github.io/talks/lens/
Докладчик - молодец, и презентация интересная. Тема линз раскрыта была. Проблема, реализация - всё было обозначено и раскрыто. Думаю добавить немного линз в свою первую поделку, избавив её от агрессивного копипаста.
Ссылка: не найдена.
Отсутствие слайдов, очень многословное повествование, размытость постановки проблемы и её решения, большое число отступлений - всё это не способствовало адекватному воспринятию доклада. Ровно та же проблема была чётко поставлена и лаконично раскрыта @tonsky здесь.
Ссылка: http://dmitriyvlasov.github.io/Presentations/review-fsharp-4.html
Вот на этом докладе хочу заострить особое внимание. Когда Lead BA начинает говорить серьёзные вещи о погромировании, это уже требует особого внимания. Респект докладчику. Как только таких Lead BA станет в 10 раз больше, так сразу я успокоюсь и подытожу: “Анализ в надёжных руках!”
По скромной оценке пришло человек 30-40. Человек 10 было в теме. Остальные пришли из Enterprise посмотреть, что же это за зверь такой, Haskell.
Не обделил вниманием митап и слабый пол
. Мне показалось, что девушки пришли туда либо со спутниками за компанию, либо же в поисках спутников. Haskell как бы ненавязчивым фоном выступил. Если это не так, то я с радостью изменю свою позицию. Вечером четверга сложил из наблюдений вполне однозначную картинку.
Тем не менее, респект всем и каждому, кто там был. Надеюсь, раз от раза качество мероприятия и количество участников будет расти. Со своей стороны постараюсь этому поспособствовать.
Да, после доклада про линзы спросил интереса ради поднять руки тех, кто из присутствующих применяет Haskell в продакшне. Ни одной руки поднято не было. Хотя в tea-break трое выловили меня и радостно сообщили, что продакшн уже знаком с Haskell…
Решил посмотреть, чем занимаются нынче фрилансеры. И наткнулся на типовую задачу, на которую ранее как-то не обращал внимания: парсеры. Требовалось реализовать парсер via HTTP, встроенный в веб-сервер со страницей отображения прогресса парсинга. Т.е. тут и трёхзвенка, и парсер. Объёмы данные большие, интересно. Думаю, почему бы и нет…
http-client
, wreq
, и hxt
, и parsec
, на котором можно написать свой парсер чего угодно. Но я остановился на tagsoup
.Итак, технологии выбраны, поехали.
Вкратце, всё просто.
import Network.HTTP as H
import Text.HTML.TagSoup
openURL :: String -> IO String
openURL x = do
y <- H.simpleHTTP (H.getRequest x)
fmap (decodeString UTF8) (H.getResponseBody y)
Company = Company
{ name :: Text
, url :: Text
, response_url :: (Maybe Text)
, city :: Text
, phone :: (Maybe Text)
, good_responses :: (Maybe Text)
} deriving Show
Tag
.forParsing
.Company
:toCompany tags = Company n u r c p gr o
where n = strip $ pack $ fromTagText $ tags !! 9
u = pack $ fromAttrib "href" $ tags !! 8
r = case (isTagOpen $ tags !! 26) of
True -> Just (pack $ fromAttrib "href" $ tags !! 26)
False -> Nothing
gr = case (isTagOpen $ tags !! 26) of
True -> Just (strip $ pack $ drop 1 $ fromTagText $ tags !! 29)
False -> Nothing
c = strip $ pack $ innerText $ takeWhile (~/= p1) $ dropWhile (~/= p2) tags
p = Just (strip $ pack $ innerText $ takeWhile (~/= p3) $ dropWhile (~/= p4) tags)
o = Nothing
p1 = "</div>" :: String
p2 = "<div class=\"b-iconed-text__text-holder\" itemprop=\"addressLocality\">" :: String
p3 = "</table>" :: String
p4 = "<table class=\"b-contact-table h-mv-10\">" :: String
И теперь, чтобы не плодить лишних параметров, можно переписать код без создания групп:
С парсером разобрались. Идём дальше.
Company
, можно использовать config/models
, тогда компания свернётся в такой код. При компиляции этот код раскрывается в нечто куда большее, подробнее можно узнать здесь:Company
name Text
url Text
response_url Text Maybe
city Text
phone Text Maybe
good_responses Text Maybe
options Text Maybe
UniqueCompany url
deriving Show
count' = do
let sql = "select count(1) cntr from category" :: Text
[(Single cntr)] :: [(Single Int)] <- runDB $ rawSql sql []
return cntr
Если колонок несколько, то тогда следует вернуть список “простых значений” ([Single a]
) и “избавиться от простоты” (unSingle
). Важно, чтобы между топовыми select
и from
не было лишних знаков вопроса, иначе запрос скомпилируется, но в рантайме будут досадные ошибки. Такова прелесть Raw SQL
. Ну а если существует необходимость конкатенировать результат запроса со строкой, содержащей символ знака вопроса (модификация url в базе), то тогда можно обернуть результат конкатенации в подзапрос, вопрос исчезнет из топа.
let sql= "select ??, ?? from category ca, company co where ca.id = co.category_id"
objs :: [(Entity Category, Entity Company)] <- runDB $ rawSql sql []
IO
, тут используется (YesodPersist site, YesodPersistBackend site ~ SqlBackend) => HandlerT site IO
. Для того, чтобы осуществлять действия с побочными эффектами, нужно использовать liftIO
. Она смещает акцент из IO в нужную монаду. Парсинг в обработчике станет выглядеть так: do tags <- liftIO $ fmap parseTags $ openURL $ unpack a
let obs = fmap (toObj c) $ sections (~== b) $ tail tags
runDB $ mapM_ insertUnique obs
cabal clean
yesod devel # cabal install