Halcyon Engine update

After an internal meeting last week we decided we are going to backport select features of Halcyon 6.4 to our current mainline. The fact is that there are a few reasons that may delay a 6.4 release:
First and most important is the numerous edge cases with older CMS's that we still need to take care.
Secondly we will be offline (Summer vacation) between the 10th and 30th of August (Our technical support will be 24/7/365 as always).

Our original plan was to have the engine detect the version of the CMS and apply filters on demand but this adds complexity which means we may not be able to release on time. Thus we decided we will be backporting some features we think are extremely important for performance of specific payloads.

One of the payloads that will get a big boost in performance is media heavy apps, in our internal testing we have on average a 38% boost in delivery of the content. which is the difference between a 3.6s to 1.4s page delivery.

Last Thursday(2018/07/05) we started pushing the release to all of our servers and since Saturday(2018/07/07) every site is been served using the latest version.

We will still try to hit our target for an August release.

Best Regards,
George Pantazis

Fun Fact: On 2018/07/03 03:19:41 UTC+3 Freca served the http request since eSGR was incorporated in 2009.

Getting Ready for Halcyon 6.4

We are in the process for finalizing the next update to Halcyon. All of our hosting servers are on 6.1 with a few exceptions so this release will be bringing everything included in 6.2 too. The current changelog is:
Structured Data Parsing
Automatic Brotli compression
Use new API for Soteria
New self healing proxy engine
Internal sitemap engine update
AMP fully Supported
http2 push fully supported
Core Changes (10%-12% speed increase)
Joomla 4.x Ready
New Wordpress Rules
Redis 4.x Redis
Switch to eSG mcache
We don't expect any additions or feature removal before its release on August.
Due to the nature of some of the changes, we will be rolling this release on newly booted VMs, this will have no effect on website uptime since they will be moved from the old VMs live. This release will make everything ready for some very interesting features coming up with 6.5/6.6 that will improve both serving speed and greater reliability to our hosting services.
Best Regards,

Switch of minimum PHP to 7.0

We would like to inform you that we have switched to PHP 7.0 as the minimum version on all shared hosting accounts. You can still use PHP 5.6 if you had it previously manually set and you can still manualy force it per account. By August 2018 we will be moving to PHP 7.1 as minimum and we expect to switch to 7.2 by February of 2019.

We have done multiple tests on all hosted services and have identifies really old apps that would fail, in those cases we automatically set the app to 5.6 and notified the user. Please note that by February 2019 we will completely remove 5.6 from all shared hosting servers.


Network issue due to storage upgrade.

After our last upgrade to our storage devices with multiple NVMe disks we run into the issue that the disk bandwidth supported overcome the capacity of our storage network connections. The new storage does 4714.3 MB/s (Read) and 4363.1 MB/s (Write) per Storage Volume. These volumes are then multiplexed per availability zone and can reach up to 21GB/s for Read and 17GB/s for Write operations. 

These new storage zones had the side effect of completely saturating the 40Gbps connections on each storage zone, during the initialisation of our storage zones this morning we managed to run in full capacity the local backbone (2x40Gbps) which resulted in minor delays in packet delivery for http(s) traffic, it was fortunate that our QoS rules were automatically applied by Halcyon and the packet loss was less than 0.7%.

Today we will be adding another 40Gbps on the storage nodes (20G per zone) and we will be ordering the needed hardware to reach the needed 80GB per zone that we require to fully support our new storage nodes, till then we will be rate limiting our storage nodes to a maximum of 7.5GB/s.

There will be no downtime on any of our services during the network maintenance and storage changes that will be performed today.


NVMe Upgrades

We have received our new NVMe disks and we will be performing upgrades on our RT(Caching & Proxy Storage) storage servers on 2018/03/07 16:00. In total we will be adding 120TB of NVMe storage (60TB usable) and ther will be no downtime to any services. About 15% of this storage will be used to provide more Caching & Proxy Storage to current customers for free.


DNS CAA Record Support

We would like to inform you we are now fully supporting CAA records in our DNS servers. You can find more information on what CAA is here Wikipedia.
For security purposes we will be creating the records for all SSL certificates we have added or will add to your services as long as the CSR is created in one of our systems. During this process we will set the iodef record to your accounts email address so that you will receive a notification if someone tries to issue a rogue certificate for your domains.

We will provide support to any of our users that would like to set those records on domains served by our DNS servers free of charge.

Best Regards,

Members Area Updates (Fixes for slow performance)

Our members area has had performance issues for quite some time. The issue is that we have been adding functionality for years now and thus we have managed to keep our users logged in performing work from one single location 24/7. This slowlly degraded performance in our members area since the servers had to communicate and await for replies from every other services through APIs. 

Today we pushed a big update that completely overhauls the underlying engine and uses are new API end-points. The new engine and API is written in pure Go which is a lot faster than using PHP & Perl. All requests now are async and the payloads are all standardized json. This change had a huge impact on performance, in its current state (no optimization have been done yet) we are serving 3 times the content on 50% of resources. the end-user result is that previous tasks that needed in excess of 20 seconds are now served in less than 3.   
For the next few months we will be gathering performance data so that we can start optimising under-performing functions as well as slowly moving from our SQL back-end to our MDS Store.
Best Regards,

Important SSL Notice: (Symantec, Thawtee, VeriSign, Equifax, GeoTrust, and RapidSSL)

From 2018/01/18 we will automatically replace any SSL certificate on customers accounts issued by Symantec, Thawtee, VeriSign, Equifax, GeoTrust, and RapidSSL. We have reached this decision since major browsers will stop trusting any SSL issued by Symantec and its subsidiaries by April 2018. 
The certificates will be replaced automatically with an equivalent from Comodo at no extra charge for the remainder of its life time. There will be no disruption of service during the change, there will be no setup fees or service charges. 
After your new Comodo certificate expires you will automatically be moved to our Free DV LE or DV Comodo certs for as long as you wish or you can purchase another certificate of your choice from a reputable CA.
If you for any reason would like to keep your old certificate you may contact us at any time between now and 2018/01/17, please note that we are setting a cut off date of 2018/03/20, after that date we will not allow you to keep that certificate active in our systems.
VPS & Dedicated Customers: We have scanned all IP's in our network already and tried to identify all domains hosted in our services, we have send you a special email informing you of the issue and our decision. You can use the included form so you can exchange your current certificates.. Again we will exchange your current certificate for free with an equivalent from Comodo, if you require assistance please open a ticket and we will perform the exchange for you for free.

Customers with our free certificates will not have any changes since their certificates are either from Comodo or Lets Encrypt which are not affected in any way.  

Further Information on why we are doing this: Google Security Blog

Best Regards,

Unplanned hardware upgrade (Meltdown & Spectre).

The recent Intel cpu bugs resulted in us doing a hardware overhaul of most of our infrastructure. Unfortunately the cpu bugs didnt only lower the security of our infrastructure but also dropped the performance significantly.

DB Cluster:
On all of our services we always work with a cpu margin of 20% that is used in emergencies or temporary expansion. On our DB cluster the aggregate of all nodes meant we had 2.6 extra servers than what was needed. Our DB cluster suffered most with an average of 42% drop in performance, which in essence meant we needed an extra 3 servers to even cope with the current requirements.

We added 3x new AMD EPYC 7401P 24/48 in the cluster that provides the same core count as 12x Xeon E5 6/12 and 75% more performance in computing power. The new cluster of only 3x new servers is at 47% load on average. We will be adding another server in mid-2018 just to increase redundancy and another one as planned in mid-2019.

Status: DB Cluster Is fully functional and ready.

In the summer of 2017 (2017/08/09) we switched all of our nodes from Xeon E5 6/12 to EPYC 7401P in a rate of 3:1. That was done purely for performance reasons, at the same time we completely elliminated magnetic storage on our VPS service and lowered the prices dramatically.

Status: VPS Service is fully functional and ready.

Shared Hosting:

In our original schedule we would have started upgrading our platform mid-2019 but the Intel bug has forced us to expedite the process. We switched all Xeon E5 dual socket servers (effective 12/24 per server) to EPYC 7601 (effective 64/128 per server), the RAM was upgraded from 128GB to 256GB. Finally we moved our hybrid shared hosting to pure SSD hosting. With this change we quintupled our cores thus octupling our computing power, doubled our memory capacity, quadrupled our disk bandwidth and octupled our iops capacity.

Status: We are in the process of starting the move of all accounts to their new home.

We will update with more information on the the bugs and mitigation issues.


Status Page Moved

We have moved the front-end of the status page.  API route for status has NOT changed.

If you use DNS SRV Records for identification you do not need to change anything otherwise please set the URL as:


Service Status

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Halcyon Update to 6.3

We are updating the server to the latest version of Halcyon (6.3)

Elasticsearch Cluster Issues

The issue has been fixed. We have started processing the backlog (approximately 9GB), we expect the process to be finished in 20 minutes.

Elasticsearch Cluster Issues

We think we have identified the issue and working on mitigating it.

Hardware Upgrade

Upgrade Finished

Hardware Upgrade

We will be upgrading radagast to 10 cores (20 Threads).

Server Reboot

Reboot required for kernel change in the host.

Elasticsearch Cluster Issues

We are investigating a number of issues with our elasticsearch cluster. At this time we are buffering logs without issues but unfortunately we can't update logs in realtime for customers. We will be exporting summaries every 5 minutes till the issue gets resolved.

Denethor Issue

We identified the issue with the proxy and

Denethor Issue

Denethor appears to have issues with the proxy, we are investigating the issue.

Container communication issue

We are noticing above average droped messages in que. We are investigating the issue.

Server issue

Server has high load issues

General Shared Hosting Maintenance

We will performing general maintenance on Shared Hosting Servers. There will be no planned downtime and the process is expected to finish in 40 minutes.

Software Upgrade

We did an upgrade of the monitoring software, no downtime was needed during the process.

SSL Issue

The issue has been fixed and we have pushed new code to all servers to make sure the issue doesn't happen again.

Contact Us

  • This email address is being protected from spambots. You need JavaScript enabled to view it.

Copyright © 2009-2018
eServices Greece