tag:status.conjured.co,2005:/historyConjured Status - Incident History2024-03-18T20:29:50-06:00Conjuredtag:status.conjured.co,2005:Incident/162959202023-02-27T14:15:34-07:002023-02-27T14:15:34-07:00Admin Outage<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>14:15</var> MST</small><br><strong>Resolved</strong> - This incident is resolved. If you are still having issues, please contact us at support@conjured.co.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>12:28</var> MST</small><br><strong>Monitoring</strong> - We are monitoring the fixes that we have put in place. Service should be back to normal, however if you are still having any issues, please contact us at support@conjured.co.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>11:29</var> MST</small><br><strong>Identified</strong> - We have identified the issue causing the Conjured Membership admin dashboard to display errors and have pushed a fix. The dashboard should be loading for most merchants now - if it is not loading for you, please contact us at support@conjured.co.<br /><br />We are still troubleshooting to ensure this issue does not happen again and are updating our servers to provide more stability in the future. Thanks so much for bearing with us.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>10:28</var> MST</small><br><strong>Investigating</strong> - We are currently investigating reports of 504 errors being displayed in the admin area of Conjured Memberships. We will update here as soon as we know more.<br /><br />Please note that this should not impact your customers' ability to purchase new memberships. If you have any additional questions, please contact us at support@conjured.co.</p>tag:status.conjured.co,2005:Incident/98378402022-04-22T21:54:43-06:002022-04-22T21:54:43-06:00Conjured Referrals Planned Maintenance Scheduled: 8:30p - 11:30p EST<p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>21:54</var> MDT</small><br><strong>Resolved</strong> - The scheduled maintenance is complete. If you have any issues, please contact us at support@conjured.co.</p><p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>19:10</var> MDT</small><br><strong>Update</strong> - We are undergoing a planned maintenance that will last no more than 3 hours beginning at 8:30p EST. Service is currently fully degraded, and will be restored by 11:30p EST. If you are experiencing any difficulties outside of the listed window, please contact us at support@conjured.co</p><p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>18:30</var> MDT</small><br><strong>Identified</strong> - We are undergoing a planned maintenance that will last no more than 3 hours beginning at 8:30p EST. Service will be fully degraded for approximately 2 hours during that period. If you are experiencing any difficulties outside of the listed window, please contact us at support@conjured.co</p>tag:status.conjured.co,2005:Incident/80028522021-09-15T15:00:34-06:002021-09-15T15:00:34-06:00Reported 503 Errors<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>15:00</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. This issue was caused by the same spike in traffic as the previous incident on September 2. We are still in the process of restructuring our server architecture to more effectively prevent downtime due to this type of massive spike. As always, if you have any questions or concerns, please contact us at support@conjured.co.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>14:30</var> MDT</small><br><strong>Monitoring</strong> - We've implemented a fix and are monitoring the situation. We will follow up with a postmortem soon.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>14:18</var> MDT</small><br><strong>Investigating</strong> - We are investigating reports of 503 errors being seen and slow-to-load customer-facing pages. We will report back here when we have more information.</p>tag:status.conjured.co,2005:Incident/79011572021-09-02T09:54:47-06:002021-09-02T09:54:47-06:00Reported Downtime & 503 Errors<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>09:54</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. <br /><br />Beginning at about 2:30am mountain time, our servers were flooded with a massive spike in traffic that resembled a DDOS attack. This brought our servers down and caused both Conjured Referrals and Conjured Subscriptions to be non-functional for both the admin dashboard and the customer-facing pages. We resolved the issue at approximately 5:30am mountain time. <br /><br />As a direct result of this issue, we are restructuring our server architecture to more effectively prevent downtime due to this type of massive spike. As always, if you have any questions or concerns, please contact us at support@conjured.co.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>05:41</var> MDT</small><br><strong>Monitoring</strong> - Service has been restored and we are monitoring the fix. Full post-mortem to come.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>05:35</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating reports that our apps are experiencing downtime and returning 503 errors. We are working as fast as possible to locate the source of the problem and restore service.</p>tag:status.conjured.co,2005:Incident/74466552021-07-08T14:53:25-06:002021-07-08T14:53:25-06:00Appearance Tab Not Displaying<p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>14:53</var> MDT</small><br><strong>Resolved</strong> - The patch appears to have solved the issue. If you are still having issues with this, or with anything else, please contact us at support@conjured.co.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>14:35</var> MDT</small><br><strong>Identified</strong> - We have identified the cause of the issue and are patching the bug now. We don't expect the downtime to be longer than a few more minutes.</p><p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>14:27</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating reports that the Appearance tab within the Campaign details is not displaying. This issue does not appear to affect any customer-facing pages and is only an issue for campaigns created prior to July 8, 2021 at 2:15p. New campaigns do not appear to be affected. We will update again when we have identified the issue.</p>tag:status.conjured.co,2005:Incident/59774712021-01-08T09:29:33-07:002021-01-08T09:29:33-07:00Conjured Referrals - Reports of Slow Load Times<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>09:29</var> MST</small><br><strong>Resolved</strong> - This issue appears to be resolved. Please contact support@conjured.co with any further issues.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>08:55</var> MST</small><br><strong>Monitoring</strong> - We believe our hosting company has addressed this issue and speeds should be back to normal. Please contact us at support@conjured.co if you are still experiencing any slow down.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>08:26</var> MST</small><br><strong>Investigating</strong> - We are currently investigating an issue that was reported of slow load times with Conjured Referrals.</p>tag:status.conjured.co,2005:Incident/56713522020-11-29T08:44:45-07:002020-11-29T08:44:45-07:00Conjured Referrals - Reports of Errors in Admin Dashboard<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>08:44</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved. If you are having any continued issues, please email us at support@conjured.co.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>10:58</var> MST</small><br><strong>Monitoring</strong> - We believe our hosting company, DigitalOcean, has finally identified the issue affecting the errors and slow speeds. It appears that the issue with the "upstream connect error or disconnect/reset before headers. reset reason: local reset" error has been resolved, and speeds are back to normal. If you are still experiencing slowdowns or errors, please contact us at support@conjured.co so we can continue troubleshooting with our hosting company.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>11:43</var> MST</small><br><strong>Investigating</strong> - We are currently investigating an issue reported by clients of an error in the Conjured Referrals dashboard:<br /><br />"upstream connect error or disconnect/reset before headers. reset reason: local reset"<br /><br />We are also investigating reports of slowness in the customer-facing referral pages. At this time, we believe it has to do with our hosting provider, DigitalOcean, and have contacted their support to get to the source of the issue. We will update here as more information becomes available.</p>tag:status.conjured.co,2005:Incident/55073912020-11-06T19:22:50-07:002020-11-06T19:22:50-07:00Navigation Issues in Conjured Referrals and Conjured Subscriptions<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>19:22</var> MST</small><br><strong>Resolved</strong> - This issue has been resolved. As always, if you're having any issues with our apps, please reach out to us at support@conjured.co</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>18:39</var> MST</small><br><strong>Monitoring</strong> - We have implemented a fix and are now monitoring the issue. If you are still having issues, please contact us at support@conjured.co</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>17:36</var> MST</small><br><strong>Identified</strong> - We have identified this issue as being related to our server's load balancer and the related proxy IP addresses. We are currently working on a fix.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>16:45</var> MST</small><br><strong>Investigating</strong> - We are currently investigating reports that navigating to other pages within our apps (for example, the "View > Campaigns" page) is causing "endless loading" where the page never displays. This also extends to modals such as the "Refund Payment" modal within Conjured Subscriptions.</p>tag:status.conjured.co,2005:Incident/55057752020-11-06T13:29:43-07:002020-11-06T13:29:43-07:00Intermittent Down Time for Conjured Referrals<p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>13:29</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved. If you are having any issues, please email support@conjured.co for additional assistance.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>12:13</var> MST</small><br><strong>Monitoring</strong> - We have scaled up our servers to fix the issue, and we are now monitoring the results.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>12:12</var> MST</small><br><strong>Investigating</strong> - We are investigating reports of intermittent downtime for Conjured Referrals.</p>tag:status.conjured.co,2005:Incident/54359722020-10-28T17:39:34-06:002020-10-28T17:39:34-06:00Scheduled Upgrade<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>17:39</var> MDT</small><br><strong>Resolved</strong> - All scheduled upgrades have been made. Emails have been restored. If you're having any issues, please contact us at support@conjured.co</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>14:40</var> MDT</small><br><strong>Update</strong> - We are upgrading mail servers, so emails will be delayed in sending by approximately 1 hour.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>12:33</var> MDT</small><br><strong>Identified</strong> - We are implementing a scheduled upgrade for all our servers and frameworks. The upgrade will be rolled out to a few servers at a time over the course of the next 1-2 days to ensure uptime and reliability. If you have any issues, please contact support@conjured.co.</p>tag:status.conjured.co,2005:Incident/45764282020-07-09T18:23:07-06:002020-07-09T18:23:07-06:00Conjured Referrals - Downtime<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>18:23</var> MDT</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>17:59</var> MDT</small><br><strong>Investigating</strong> - We are investigating reports of intermittent 500 errors by clients of Conjured Referrals.</p>tag:status.conjured.co,2005:Incident/45456242020-07-06T09:01:05-06:002020-07-06T09:04:07-06:00Conjured Subscriptions Downtime<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>09:01</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved. Please read the incident's postmortem for more details.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>08:19</var> MDT</small><br><strong>Monitoring</strong> - We've issued a fix and are monitoring the situation. We'll be issuing a postmortem on the issue shortly.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>07:55</var> MDT</small><br><strong>Identified</strong> - We have identified the issue as relating to our Redis database, which is hosted by DigitalOcean and provides session and caching, being down. We are actively working to fix this and should have a solution shortly.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>07:49</var> MDT</small><br><strong>Investigating</strong> - We're currently investigating reports of downtime in the Conjured Subscriptions app.</p>tag:status.conjured.co,2005:Incident/45396222020-07-05T15:12:51-06:002020-07-06T09:03:31-06:00Conjured Referrals Intermittent Downtime<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>15:12</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>14:49</var> MDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>14:38</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating an issue that is causing 500 errors to show in the admin dashboard, and is affecting referral page display.</p>tag:status.conjured.co,2005:Incident/35082392020-02-03T13:24:30-07:002020-02-03T13:24:30-07:00Service Degradation<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>13:24</var> MST</small><br><strong>Resolved</strong> - This incident has been resolved. Please don't hesitate to reach out if you have any issues - support@conjured.co.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:46</var> MST</small><br><strong>Monitoring</strong> - We have finally pin-pointed the source of the traffic spike and have blocked the offending requests. All services should be operational. We are monitoring the patch, and are working on a solution to prevent this from occurring in the future. If you need any additional assistance, or are continuing to experience outages or slow-downs, please contact our support team at support@conjured.co so we can help troubleshoot.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>22:03</var> MST</small><br><strong>Identified</strong> - We now believe this issue is due to a DDOS attack on our servers, which have experienced a sudden and unprecedented spike in requests. <br /><br />We have worked to partition Conjured Subscriptions to minimize downtime and slow load times, however Conjured Referrals is still experiencing substantially degraded performance. <br /><br />We are continuing to work aggressively to find a solution to the outage and will continue to update here as we learn more.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>16:35</var> MST</small><br><strong>Investigating</strong> - We are currently investigating reports of service degradation and slow load times with both Conjured Referrals and Conjured Subscriptions. We are investigating the source of the degradation (and whether it may have to do with the issues currently being investigated by DigitalOcean (https://status.digitalocean.com/).</p>tag:status.conjured.co,2005:Incident/30463872019-10-17T11:07:17-06:002019-10-17T11:07:17-06:00Minor Bug in Subscription Discount Codes<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:07</var> MDT</small><br><strong>Resolved</strong> - We have just pushed a fix that will ensure that the discount value is taken from the highest priced subscription in the cart so the full amount of the discount can be used.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>11:03</var> MDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>11:03</var> MDT</small><br><strong>Update</strong> - We have pushed a fix for the issue of orders being created in Shopify with the incorrect value for the line item in the order (#2). Still working on a fix for the customer not being discounted the full amount of the discount code when charged (#1).</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>09:19</var> MDT</small><br><strong>Identified</strong> - We have identified a bug in the way discount codes are handled for subscriptions when the following conditions are true:<br /><br />a. The discount code applies to an order containing "Any Subscription Box",<br />b. The customer orders more than one subscription during checkout, and<br />c. The value of the discount is greater than the value of the first subscription box added to the cart.<br /><br />In this case, the first subscription box is discounted 100% (since it costs less than the value of the discount), but the remainder of the discount amount isn't applied to the rest of the cart. This causes two problems:<br /><br />1. The customer is not being discounted the full amount of the discount code when charged, and<br />2. The order created in Shopify has an incorrect value for the line items in the order.<br /><br />We are currently in the process of patching this bug and will update here when a fix is live.</p>tag:status.conjured.co,2005:Incident/30224172019-09-21T10:13:16-06:002019-09-21T10:13:16-06:00Hosted PostgreSQL Slowness<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>10:13</var> MDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>09:51</var> MDT</small><br><strong>Identified</strong> - We are in the process of upgrading the PostgreSQL hosted instances that are affected and streamlining certain queries to improve speed and performance.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>09:35</var> MDT</small><br><strong>Investigating</strong> - We are currently investigating a slowdown in the some hosted PostgreSQL instances. This may affect app load time and in some cases may be causing timeouts.</p>tag:status.conjured.co,2005:Incident/28962982019-08-21T11:03:04-06:002019-08-21T11:03:04-06:00Conjured Subscriptions - Braintree Payments Timeout<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>11:03</var> MDT</small><br><strong>Resolved</strong> - We can confirm this incident has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>10:04</var> MDT</small><br><strong>Monitoring</strong> - Braintree's API appears to have returned to normal, but we will continue monitoring to ensure the issue has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>09:59</var> MDT</small><br><strong>Investigating</strong> - We are investigating issues where Conjured Subscriptions is receiving a "Braintree\Exception\Timeout" in response to API calls to Braintree. This may interfere with customers' ability to checkout when Braintree is being used as a payment processor. We will update further when we have additional information.</p>tag:status.conjured.co,2005:Incident/28738332019-08-16T15:30:02-06:002019-08-16T15:33:01-06:00Conjured Subscriptions - Customer-selected SKUs being replaced by default in 1st shipment<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>15:30</var> MDT</small><br><strong>Resolved</strong> - A fix has been issued for this bug. Please contact support@conjured.co with any questions or concerns.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>15:26</var> MDT</small><br><strong>Identified</strong> - We are currently investigating an issue where SKUs selected by a customer at checkout are being replaced by the default in their Cycle 1 shipment. This incident only affects stores allowing their customers to select the product(s) included in their first shipment.</p>tag:status.conjured.co,2005:Incident/28732102019-08-11T15:06:43-06:002019-08-16T12:45:38-06:00Conjured Subscriptions - Checkout Downtime<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>15:06</var> MDT</small><br><strong>Resolved</strong> - Due to an upgrade in our Load Balancer's Proxy settings, customers were unable to checkout. Incident was reported at around noon mountain time and solved at 3:06p mountain time.</p>