PostgreSQL on Hadoop

uk: Комп'ютерні науки

description: Bridging the Divide with Distributed Foreign Tables Apache Hadoop is an open-source framework that enables the construction of distributed, data-intensive applications running on clusters of commodity hardware. Building on a foundation initially composed of the MapReduce programming model and Hadoop Distributed Filesystem, in recent years Hadoop has expanded to include applications for data warehousing (Apache Hive), ETL (Apache Pig), and NoSQL column stores (Apache HBase). In this talk we describe recent work done at Citus Data that makes it possible to run a distributed version of PostgreSQL on top of Hadoop in a manner that combines the rich feature set and low-latency responsiveness of PostgreSQL with the scalability and performance characteristics of Hadoop. This talk will begin with a high level overview of Hadoop that focuses on its distributed storage layer and block-based replication model. Next we will look at the data model of the Apache Hive data warehousing system and explain how it enables features such as schema-on-read, support for semi-structured data, and pluggable storage formats. Finally, we will describe how we leveraged these ideas and Foreign Data Wrappers to build a distributed version of PostgreSQL. This version runs natively on Hadoop clusters and seamlessly integrates with other components in the Hadoop ecosystem.

uk: вхід без авторизації


Query Planning Gone Wrong

Robert Haas

type: Person

cs: Informatika

de: Informatik

fi: tietojenkäsittelytiede

uk: Комп'ютерні науки

en: Computer Science

fr: Informatique

da: Datalogi

es: Ciencias de la Computación

nl: Informatica

cs: Technické vědy

de: Ingenieurwissenschaften

fi: tekniikka

uk: Інженерія

en: Engineering Sciences

fr: Ingénierie

da: Ingeniørvidenskab

es: Ingeniería

nl: Engineering

description: An analysis of common query planner failure modes The PostgreSQL query planner does an excellent job with most queries, but no query planner is perfect. In this talk, I'll analyze some of the commonly-reported cases where it fails to produce an acceptable plan, based on posts to pgsql-performance and other cases I've encountered on the job.

TIB AV-Portal

type: Service

type: WebContent

type: MediaObject

@language: en

cs: přihlášení není vyžadováno

de: keine Anmeldung erforderlich

uk: вхід без авторизації

en: no login required

nl: geen login vereist

name: PGCon - PostgreSQL Conference for Users and Developers, Andrea Ross

type: Organization

cs: Video

de: Video

fi: Video

uk: Відео

en: Video

fr: Vidéo

da: Video

nl: Video

es: Video


Row Level Security

uk: Комп'ютерні науки

description: Row-level security has a feature with longstanding development. It can enforce users to reference or modify part of rows according to the configured security policy. So, we can utilize this feature as if virtual private database on other commercial database, however, we designed this feature much carefully to keep both of reliable security and minimum performance trade-off. In addition, we enhanced usability to allow row-level security policy using usual expressions, also plan to integration with label based mandatory access control. This session introduces which was the problematic scenario being called "leaky-view", solutions for them as basis of this feature, row-level security feature being newly supported, and the future plan towards integration with label based mandatory access control. We don't assume audience has deep knowledge on planner. Typical use cases will be helpful for web-application developers who like to ensure reliable separation between users.

uk: вхід без авторизації


PostgreSQL and Sphinx

uk: Комп'ютерні науки

description: Full Text Search extension How to integrate both tools and obtain the best performance + reliability. This talk is focused on the new and hottest features on Sphinx (2.1.1 beta) and PostgreSQL. How to combine those tools and HA new features will be showed up during the presentation and also, how to reach a high performance and simple text search

uk: вхід без авторизації


SQL Hints, Tips, Tricks, and Tuning

uk: Комп'ютерні науки

description: What is the difference between inner, outer, cross and natural join? What is a correlated subselect? What is the advantage and what the disadvantage of an index. Is NOT and OR evil? When do I need normalisation and when denormalisation? All this questions and lots of other performance tips and tricks on SQL level will be given in the talk.

uk: вхід без авторизації


Billion Tables Project (BTP)

uk: Комп'ютерні науки

description: Aka how long a "\dt" takes on a 1B tables database Usually "large" databases are considered as such for the high number of records they hold, reaching billions or even more than that. But what about creating a billion... tables? Sometime ago, this apparently crazy question was found in a database soup. It may not be your day-to-day task, but the task of creating them exposes some topics about PostgreSQL internals, performance and large databases that may be really worth for your day-to-day. Join us for this talk, where we'll be discussing topics such as catalogue structure and storage requirements, table speed creation, differences between PostgreSQL versions and durability vs. table creation speed tradeoffs, among others. And, of course, how long a "\dt" takes on a 1B tables database :) This talk will explore all the steps taken to achieve such a result, raising questions on topics such as: The catalogue structure and its storage requirements. Table creation speed. Durability tradeoffs to achieve the desired goal. Strategy to be able to create the 1B tables. Scripts / programs used. How the database behaves under such a high table count. Differences in table creation speed and other shortcuts between different PostgreSQL versions. How the storage media and database memory affects the table creation speed and the feasibility of the task. If it makes sense to have such a database. It is intended to be a funny, open talk, for a beginner to medium level audience, interested in large databases, performance and PostgreSQL internals.

uk: вхід без авторизації


Understanding PostgreSQL timelines

uk: Комп'ютерні науки

description: Whenever you perform point-in-time-recovery or failover in PostgreSQL, a new timeline is created. In the best case, timelines help you stay sane in complicated recovery situations and keep your database consistent. In the worst case, they will thoroughly confuse you. This talk explains timelines, how they are used by the system, and how you can stay sane with them and even take advantage of them. Version 9.3 brings some improvements to the handling of timelines, these are discussed as well.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: If you want to know right now what your server has been doing When problems arise, every DBA wants to know what the database has been up to. If your best idea so far has been to parse the log files or load the CSV logs into another database, you might want to learn about a better way. With logging hooks, you can attach data sinks directly to the PostgreSQL server log stream and analyze logging events as they happen. Many useful applications arise this way: put log data into another PostgreSQL database for ad hoc querying store log data in Hadoop for analytics throw log data into a graphing system generate alerts directly from logging events I'll show you how to put these things together and use them in practice.

uk: вхід без авторизації

name: Using logging hooks for real-time log analysis


Using PostgreSQL with Redis

uk: Комп'ютерні науки

description: Using native wrappers and a Foreign Data Wrapper for two-way Redis integration Redis is a high performance in-memory distributed data store, which can work well in conjunction with PostgreSQL in certain types of applications. The talk will introduce native wrappers for the Redis API, which allow pushing data into Redis at high speed, as well as use of both the native API and the Redis Foreign Data Wrapper to pull data from Redis. We will also discuss possible scenarios where use of each of these is a win, and the trade-offs involved.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: Logical Replication in the context of postgres to this date consists out of several independent out-of-core solutions. While some of these solutions are great, the existance of many of those also causes problems like code duplication, lack of trust, features, reliability and peformance. As part of a proposal to include one logical replication solution into core postgresql we submitted the changeset generation/extraction part as a core infrastructure to postgres. In a way its usable by all the existing replication solutions and for lots of other usecases. This talk is about: the architecture of the committed/proposed changeset generation mechanism (2/3) An overview over further proposed patches (9.4+) to get a whole logical replication into core postgres (1/3)

uk: вхід без авторизації

name: On the road to in-core logical replication!? Part 1: Architecture of Changeset Extraction


Visualizing Postgres in realtime

uk: Комп'ютерні науки

description: Postgres comes with several introspection tools out of the box. Some are easier to understand than others, but they are all useful. Recent improvements in 9.2's pgstatstatements make it even easier gain insights into the performance of your application. This talk will explore these built in tools, and what it takes to combine them to provide real-time visualizations of your database. Other topics will incldue - What metrics are the most valuable and how to use them - A deep dive into example application for realtime postgres visibility - Storage of postgres statistics - Tools for time series visualization - Collecting metrics at scale - What works and, just as importantly, what does not work

uk: вхід без авторизації


uk: Комп'ютерні науки

description: How 9.4 will shake your world Join us for this first look at new and upcoming features in the 9.4 release. We'll discuss some of the what and why of the new features, and help you start planning your upgrades. The Postgres development should be mostly through feature freeze by the time PGCon rolls around, so this talk should be able to give you a good idea of what will be in the next release of Postgres. Improvements in JSON, replication, updateable views, materialized views, and more. Hope to see you there!

uk: вхід без авторизації

name: 9.4 On The Floor: A Talk about Upcoming Features in Postgres 9-4


uk: Комп'ютерні науки

description: In this presentation I'll talk about two migrations from Oracle to PostgreSQL - the first back in 2001, which is how I got into PostgreSQL in the first place, and the second in 2013 involving the second largest stock exchange in Japan. One of the hotter PostgreSQL topics recently is the increasing trend towards enterprise-scale migrations from Oracle and other commercial databases to PostgreSQL - not only for cost savings but also for its rich and expanding feature set combined with a solid reputation for reliability. But it's hardly a new trend - that's how I first came across PostgreSQL back in 2001. I'll give a brief and hopefully humourous description of what I did back then (with some pictures of cute animals) and how it led to the Japanese stock market. Which is where I ended up in charge of the database underpinning the back office of Japan's second largest stock exchange, SBI Japannext. Since late last summer this has been powered by PostgreSQL and the changeover from Oracle has gone very smoothly. The main part of this presentation will be about how this migration was planned and implemented, and the foundations it has created for further adoption of PostgreSQL.

uk: вхід без авторизації

name: A Tale of Two Oracle Migrations: Deploying PostgreSQL at Japan's second-largest stock exchange


uk: Комп'ютерні науки

description: A case study on the trials of Emma's performance when implementing the Entity-Attribute-Value data model on their PostgreSQL database systems. Emma, Inc. is an email marketing company that provides a Web based application for managing email lists and marketing campaigns. This is a tale about Emma's experience with a database schema design refactor that is now using the Entity-Attribute-Value (EAV) data model on the members in email lists. We will briefly describe the previous data model and why we had to move to a new one. This is followed by all the places where Emma stumbled and recovered in implementing the EAV data model and rebuilding the application around it. Finally we will touch on where Emma might go next with respect to the data.

uk: вхід без авторизації

name: An Adventure in Data Modeling: The Entity-Attribute-Value Data Model


uk: Комп'ютерні науки

description: by Greg Smith The easiest type of benchmark to run on a database is checking its throughput: the total transactions processed during some period of time. But throughput isn't what application users care about. They want low latency. Latency and throughput have a complex dependency on each other, and you'll need a whole new type of test to balance them well. Recent advances in benchmarking tools, like the rate limit in pgbench 9.4, make it possible to analyze latency in a way that highlights this difficult to see area. The graphics and reports of the pgbench-tools package make it possible to see how tuning changes impact both latency and throughput. That that lets you test the "lore" for how best to tune your PostgreSQL server to find out what really works. Using these new tools, this talk will look at the tricky correlation between throughput and latency and how to measure each usefully in common situations. We'll look at three different storage stacks with very different latency profiles: regular disk, disk with battery-backed write cache, and SSD. We'll then look at exactly how checkpoint spikes play out on each and what you can do about them. You'll never trust a simple transactions per second count again!

uk: вхід без авторизації

name: Balancing Database Storage Latency And Throughput: PostgreSQL for Secure Enterprises


uk: Комп'ютерні науки

description: A tour of the operator class facility A PostgreSQL operator class tells the database server how to sort, match, and index values of a particular data type, and an operator family ties related data types together for the purpose of these operations. They constitute agreements between data type authors and data type-independent code, with most data types defining at least one operator class. This talk will explore the theory behind operator classes and families, including the assumptions data type-independent code is and is not entitled to make based on available operator classes. We will walk through the creation of new operator classes, some practical and others deliberately perverse, and examine some exceptional operator classes already present in PostgreSQL. Writing a query expression such as "val greater than = $1" without knowing the data type of "val" is almost always a mistake. It will work for the vast majority of data types, but it will malfunction in rare cases. We will examine the use of operator classes to adapt such an expression to each data type. This talk caters to writers of code intended to deal with unforeseen data types. Application code such as replication systems and client interfaces are affected, as are backend features such as integrity constraints and join algorithms. As an author of such software, you will leave the talk equipped to audit for mistakes and substitute code that harnesses operator classes and families to process values of arbitrary type. The talk will also help prospective authors of new data types.

uk: вхід без авторизації

name: I know greater-than-or-equal-to when I see it!


Explaining the Postgres Query Optimizer

uk: Комп'ютерні науки

description: The optimizer is the "brain" of the database, interpreting SQL queries and determining the fastest method of execution. This talk uses the EXPLAIN command to show how the optimizer interprets queries and determines optimal execution. Examples include scan methods, index selection, join types, and how ANALYZE statistics influence their selection. The talk will assist developers and administrators in understanding how Postgres optimally executes their queries and what steps they can take to understand and perhaps improve its behavior.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: Mapping a billion stars throughout our Galaxy and beyond Gaia is a cornerstone European Space Agency mission that will create an extraordinarily precise three-dimensional map of more than a billion stars throughout our Galaxy and beyond, mapping their motions, luminosity, temperature and composition. This huge stellar census will provide the data needed to tackle an enormous range of important problems related to the origin, structure and evolutionary history of our Galaxy. I will describe how Variability Studies of light sources observed by Gaia pose Big Data problem that we are trying to solve with the help of Postgres-XC.

uk: вхід без авторизації

name: Gaia Variability Studies: case for Postgres-XC.


HowTo DR planning for the worst

uk: Комп'ютерні науки

description: planning for the worst There's a lot more to disaster recovery than making backups. Most of DR, in fact, is planning instead of code: knowing what you need to do when disaster strikes, how to do it, and who does it. Further complicating things, management and admins are fond of preparing for unlikely events while failing to prepare for probable outages at all. There's a lot more to disaster recovery than making backups. Most of DR, in fact, is planning instead of code: knowing what you need to do when disaster strikes, how to do it, and who does it. Further complicating things, management and admins are fond of preparing for unlikely events while failing to prepare for probable outages at all. This talk will outline how to make a disaster recovery plan, and some basic dos and don'ts of DR. Included: The three most common downtime causes Determining acceptable losses (and getting management to agree) Backup vs. Replication Planning for the unexpected Against Improvising (always have a Plan B) Public Cloud DR Other Dos and Don'ts When disaster strikes, it's too late to start planning. Do it now.

uk: вхід без авторизації


PostgreSQL 9.4 and Json

uk: Комп'ютерні науки

description: We'll cover the new Json generation functions, as well as jsonb, and when to use json, jsonb or hstore. We'll end by looking into the future to see what other json functionality might be coming, such as json mutation functions. The talk will present a survey of the current and coming state of json features, as well as a speculative look at what which be in the future beyond 9.4 Overview: * the new json generation functions, such as jsonbuildobject and jsonbuildarray and how they can be used to build up arbitrarily complex and non-rectangular json. * json_typeof() * jsonb, and its relation to json and nested hstore. * how jsonb is different from json, (e.g. object field ordering, treatment of duplicate field names and whitespace) * performance differences between json and jsonb * what's missing from json/jsonb? What should we work on for 9.5? Mutation functions have been suggested, Are there others?

uk: вхід без авторизації


Implementing Parallelism in PostgreSQL

uk: Комп'ютерні науки

description: Where We Are Today, and What's On The Horizon PostgreSQL's architecture is based heavily on the idea that each connection is served by a single backend process, but CPU core counts are rising much faster than CPU speeds, and large data sets can't be efficiently processed serially. Adding parallelism to PostgreSQL requires significant architectural changes to many areas of the system, including background workers, shared memory, memory allocation, locking, GUC, transactions, snapshots, and more. In this talk, I'll give an overview of the changes made to background workers in PostgreSQL 9.4 and the new dynamic shared memory facility, which I believe will form the foundations of parallelism in PostgreSQL, and discuss some lessons I learned while implementing these features. I'll also discuss what I believe is needed next: easy allocation of dynamic shared memory, state sharing between multiple backends, lock manager improvements, and parallel algorithms; and highlight what I believe to be the key challenges in each area.

uk: вхід без авторизації


In-Memory Columnar Store for PostgreSQL

uk: Комп'ютерні науки

description: IMCS is In-Memory Columnar Store for PostgreSQL. Vertical data model is more efficient for analytic queries performing operations on entire column. IMCS provides 10-100 times improvement in performance comparing with standard SQL queries because of: data skipping: fetching only data needed for query execution parallel execution: using multiple threads to execute query vector operations: minimizing interpretation overhead and allowing SIMD instructions reduced locking overhead: simple array level locking no disk IO: all data is in memory IMCS is implemented as standard PostgreSQL extension. It provides set of function and operators for manipulations with timeseries. Some of them are analog of standard SQL operators (arithmetic, comparisons, sorting, aggregation...). But there are also complex analytic operators like calculation of ranks, percentiles, cross points and extended set of aggregates for financial application like split-adjusted price, volume-weighted average price, moving average... Columnar store manager stores data tables as sections of columns of data rather than as rows of data.In comparison, most relational DBMSs store data in rows. Such approach allows to load the whole record using one read operation which leads to better performance for OLTP queries. But OLAP queries are mostly performing operations on entire columns, for example calculating sum or average of some column. In this case vertical data representation is more efficient. Columnar store or vertical representation of data allows to achieve better performance in comparison with horizontal representation due to three factors: * Data skipping. Only columns involved in query are accessed. * Vector operations. Applying an operator to set of values minimize interpretation cost. Also SIMD instructions of modern processors accelerate execution of vector operations. Compression of data. For example such simple compression algorithm like RLE allows not only to reduce used space, but also minimize number of performed operations. IMCS is first of all oriented on work with timeseries. Timeseries is sequence of usually small fixed size elements ordered by some timestamp. Operations with timeseries rarely access some particular timeseries element, instead of it them operate either with whole timeseries either with some time interval. Such specific of timeseries operation requires special index for timeseries, which is different from traditional database indexes. Such index should not provide efficient way of locating arbitrary timeseries element. Instead of it this index should be able to efficiently extract range of timeseries elements. Advantages of IMCS approach: Fast execution based on vector operations Parallel execution of query No changes in PostgreSQL core (just standard extension) No MVCC overhead (MURSIW isolation level) No disk IO (in-memory store) Optimized for timeseries (massive operations with time slices)

uk: вхід без авторизації


PostgreSQL, Rust, and Range Types

uk: Комп'ютерні науки

description: Combining database features with language features Range Types, introduced in PostgreSQL 9.2, are an important feature for scheduling and other applications. Range Types allow much more efficient queries on ranges (e.g. ranges of time) such as "overlaps" and "contains", are less error-prone, and offer the critical "non-overlapping" constraint (useful for preventing schedule conflicts). But to make use of any advanced database features, good support in the client language is crucial. Many people use libraries to translate between database types and types in the client language -- not only is that convenient, but it contributes to safety and reduces the chance for simple errors. Rust is a powerful new language with a lot to offer, including a postgres driver that supports Range Types as first-class types. I'll be discussing how rust libraries help bridge the gap between advanced database features and practical application development. The speed of development in PostgreSQL can often outpace the ecosystem surrounding it. Performance-boosting features are used as quickly as administrators upgrade, but extensions to the SQL language take more time. Without support in the client language, SQL language features are only used by early-adopters who are willing to put up with the rough edges. As the primary developer for Range Types, I feel that their utility would be greatly increased with greater accessibility for simple applications. The problem they solve - largely the problem of scheduling - is prevalent in such applications. Rust makes a great example of how to better integrate data types into the client language. It's got Option types, which are great for handling edge cases and special values (like an empty or unbounded range); it's fast enough that the driver can be written in rust, which avoids the need for a dependency on libpq-dev and enables a little more creativity; and it's got a great community. In particular, I'd like to credit Steven Fackler who wrote the native rust-postgres driver, as well as the Range Types support! This presentation does not assume any prior knowledge of the rust language

uk: вхід без авторизації


SQLite: Protégé of PostgreSQL

uk: Комп'ютерні науки

description: The Keynote for PGCon 2014 by D. Richard Hipp SQLite has become the most widely deployed SQL database engine in the world, with over two billion installations providing SQL database services for cellphones, web-browsers, cameras, and countless other gadgets and applications. But SQLite is not in competition with PostgreSQL. Rather, PostgreSQL and SQLite complement each other, with each engine targeting a different class of problems. SQLite can be thought of as a derivative of PostgreSQL. SQLite was originally written from PostgreSQL 6.5 documentation, and the SQLite developers still use PostgreSQL as a reference platform to verify that SQLite is working correctly.

uk: вхід без авторизації


When PostgreSQL Can't, You Can

uk: Комп'ютерні науки

description: After using PostgreSQL for a while, you realize that there are missing features that would make it significantly easier to use in large production environments. Thankfully, it's extremely easy to make add-ons to enable some of those features right now without even knowing a single line of C! After using PostgreSQL for a while, you realize that there are missing features that would make it significantly easier to use in large production environments. Thankfully, it's extremely easy to make add-ons to enable some of those features right now. And you don't even have to know a single line of C code! Over the past view years I've been developing tools that have made PostgreSQL administration for our clients much easier. Table partitioning is one of the best methods for providing query performance improvements on large tables in PostgreSQL. While the documentation gives instructions on how this can be set up, it's still a very manual process. For time/serial based partitioning, the maintenance of creating new partitions is huge part of that process. PG Partition Manager is an a extension that aims to provide easy setup & maintenance for the most common partitioning types. It also provides advanced features for taking advantage of constraint exclusion and retention to drop unneeded tables. Another area where we ran into limitations was when one of our clients needed an easy method to keep the database schema checked into version control. The pg_extractor tool grew from this and has become a popular method of finely tuning the extraction of database objects from PostgreSQL into individual files. Autonomous transactions is another one of the more popular features in other RDMS systems that is missing from PostgreSQL. Begin able to reliably log a function's run status requires that function failure not roll back the entire transaction and erase entries made to the log table. PG Job Monitor takes advantage the dblink contrib module to give the behavior of an autonomous transaction, providing logging & monitoring for mission critical functions. PostgreSQL's built in replication is all or nothing, but sometimes you just need to replicate specific pieces of your database. There are several 3rd-party systems capable of accomplishing this, but their setup can been daunting to most new users and overkill if you just need to grab a few tables. Mimeo is a logical replication tool that requires minimal permissions and setup and provides several specialized means of replicating from one or many databases to a single destination. This talk will discuss these project and hopefully bring insight on how easy it is to contribute to the PostgreSQL community.

uk: вхід без авторизації


Multicorn: writing FDWs in Python

uk: Комп'ютерні науки

description: A tutorial showing off Multicorn's features Multicorn is a generic Foreign Data Wrapper which goal is to simplify development of FDWs by writing them in Python. We will see: what is an FDW what Multicorn is trying to solve how to use it, with a brief tour of the FDWs shipping with Multicorn. how to write your own FDW in python, including the new 9.3 write API the internals: what Multicorn is doing for you behind the scenes, and what it doesn't After a presentation of FDWs in general, and what the Multicorn extension really is, we will take a look at some of the FDWs bundled with Multicorn. Then, a complete tour of the Multicorn API will teach you how to write a FDW in python, including the following features: using the table definition WHERE clauses push-down output columns restrictions influencing the planner writing to a foreign table transaction management This will be a hands-on explanation, with code snippets allowing you to build your own FDW in python from scratch.

uk: вхід без авторизації


Scaling a Cloud Based Analytics Engine

uk: Комп'ютерні науки

description: Scaling a 100% Cloud Native Analytics Engine Your mission, should you chose to accept it, create a data storage system that can handle 200 Gigs of data per day on cloud servers with heavy analytics. GO! The architecture plan of a real time logging system built to handle 200g/day of data and hand it off from mid-term OLTP storage into and OLAP postgres data warehouse. It was built with heavy reliance on inheritance, dblink, and streaming replication.

uk: вхід без авторизації


Mobile Canvassing with PostgreSQL

uk: Комп'ютерні науки

description: At Moonshadow Mobile, we visualize geospatial data in a custom geospatial engine and analysis framework inside a browser. This is coupled with Ground Game, a mobile canvassing solution that lets managers and canvassers easily mount a campaign. At the heart of this solution we need a featureful database with a high level of data integrity to quickly store and retrieve customer data, all of which PostgreSQL provides. Our Ground Game mobile canvassing solution is available for iDevices, Android devices, and any device with a good mobile web browser.Campaign managers assign walking lists and questionnaires to canvassers in the field from our web application, who receive the data on their mobile devices and fill out questionnaires while going door-to-door. Data can be collected even if no connection is available and the data is synced as soon as a wifi or mobile data is available. On a typical day hundreds of canvassers around the country upload thousands of questionnaires. Our base data consists of the voter files that are collected from states and counties, augmented with psychographic information from commercial sources. At any time voter files,phone numbers,addresses,and emails are updated. Clients augment our data with their own information about donations and membership information, along with mobile canvassing. All these data streams need to be processed without causing any downtime for the campaign managers and the canvassers in the field.PostgreSQL plays a very important role in our technology stack. All canvassing data,walking lists,canvassing users,assignments,and questionnaires are stored in PostgreSQL. We also use PostGIS for the creation of the data and the assignment of voters to district zones.In this talk, I will give an overview of our architecture, and focus on how PostgreSQL has helped us deliver our product. These reasons include custom features of PostgreSQL such as bytea,json,and custom types, addon modules such as crypto and ltree,and custom functions in languages such as plperlu,along with WAL replication,hot backups,all of which give us security,peace of mind,and the ability to more rapidly develop features. Finally, we look to the future and eagerly await integrating new and additional PostgreSQL features into our workflow.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: 9.4 saw a great deal of development around a feature (now) called changeset extraction. This talk will explain what the feature does, which areas of use we see, what the state of the feature in 9.4 is and which additional features around it we want to see in future releases of postgres. Usecases for the changeset extraction feature are: Replication Solutions Auditing Cache Invalidation Federation ... Changeset extraction is the ability to extract a consistent stream of changes in the order they happened - which is very useful for replication, auditing among other things. But since that's a fairly abstract explanation, how about a short example? -- create a new changestream postgres=# SELECT * FROM create_decoding_replication_slot('slot', 'test_decoding'); slotname | xlog_position ----------+--------------- slot | 0/477D2398 (1 row) -- perform some DML postgres=# INSERT INTO replication_example(data) VALUES('somedata'); INSERT 0 1 -- and now, display all the changes postgres=# SELECT * FROM decoding_slot_get_changes('slot', 'now', 'include-timestamp', 'yes'); location | xid | data ------------+---------+--------------------------------------------------------------------- 0/477D2510 | 1484040 | BEGIN 1484040 0/477D2628 | 1484040 | table "replication_example": INSERT: id[int4]:1 data[text]:somedata 0/477D2628 | 1484040 | COMMIT 1484040 (at 2014-01-20 01:18:49.901553+01) (3 rows) All this works with a low overhead and a configurable output format.

uk: вхід без авторизації

name: SELECT * FROM changes; - Part 2: Logical Decoding


PGCon 2014: Lightning talks

name: PGCon - PostgreSQL Conference for Users and Developers

uk: Комп'ютерні науки

description: Various speakers & subjects. Database Appliance based on PostgreSQL OHAI, My name is Chelnik! BDR - Bi Directional Replication What the XL?

uk: вхід без авторизації


uk: Комп'ютерні науки

description: PgREST turns PostgreSQL into a RESTful JSON document store, allows running JavaScript and npm modules as stored procedures for queries and triggers from within the database. It also provides Firebase-compatible API for developing real-time applications. PgREST is... a JSON document store running inside PostgreSQL working with existing relational data capable of loading Node.js modules compatible with MongoLab's REST API and Firebase's real-time API This talk will cover: the building blocks of PgREST: PostgreSQL, plv8js, plv8x examples for turning existing relational data into REST endpoints building real-time applications with PgREST

uk: вхід без авторизації

name: PgREST: PostgreSQL, JavaScript, and REST


Postgres in Amazon RDS

uk: Комп'ютерні науки

description: A compelling option for startups In recent years, Amazon's cloud service is key for deploying and scaling IT infrastructure tiers, and this is not different with Postgres as well. Amazon's relational database service for Postgres RDS - is a new but compelling option for the startups, when the resources are scarce and administrators are hard to find! RDS is a great option for people to achieve in terms of rapid deployment of replicas, ease of failover, and the ability to easily redeploy hosts when failures occur, rather than spending extensive time trying to repair. In the talk, I'm planning to topics related to managing Postgres RDS : Introduction of RDS Benefits of using RDS Challenges with RDS Choosing EC2 vs RDS Security Tuning Scaling Monitoring Backups Cost Generic Recommendations

uk: вхід без авторизації


uk: Комп'ютерні науки

description: While it usually Just Works, sometimes it's useful to know exactly what happens between hitting return in psql and seeing results appear on the screen. This talk will explain how the PostgreSQL frontend/backend protocol works. We will look at the low-level blocks from which the protocol is built and try to give the audience a solid understanding of how bytes are transferred back and forth between their applications and the Postgres server. Even if you never directly deal with the PostgreSQL protocol, every application you're writing or maintaining uses it and it's still useful to know the basics of how it works. Sometimes, knowing the fundamentals can also help you understand some behaviour that otherwise would seems mysterious or quirky. The talk is aimed at users, DBAs and system administrators interested in learning a bit about how the sausage gets sent. We'll go through: protocol versions and how they differ differences between simple and extended protocol text and binary query formats authentication and encryption asynchronous features: async queries and NOTICE the COPY protocol query cancellation and how does it work future development of the protocol

uk: вхід без авторизації

name: Postgres on the wire: A look at the PostgreSQL wire protocol


uk: Комп'ютерні науки

description: In 2010 Russian State Duma (lower chamber of parliament) renewed its web site. That was the first step in modernization towards open data and e-goverment in further years. PostgreSQL took a central place in data integration between intranet systems and open web services. During recent years many new open data systems were launched powered by PostgreSQL. This talk covers the issues of PostgreSQL integration into State Duma intranet systems, which were mostly running Oracle Database as well as implementation of many open data systems. Data between intranet systems and PostgreSQL databases are mostly synchronized using ora2pg. Full synchronization was used for small databases as well as incremental synchronization for large databases. Open data systems implemented on PostgreSQL including: Law and draft law search systems Open vote system Verbatim system Using PostgreSQL database allows to save license const as well as achieve great performance and features.

uk: вхід без авторизації

name: PostgreSQL experience in Russian Parliament: Success story of PostgreSQL in Russian State Duma


uk: Комп'ютерні науки

description: It is hard to maintain complex C code free of bugs. Numerous tools have been developed to help, and we have thrown many of them at the PostgreSQL source code. We will take a look at tools such as Coverity Clang scan-build AddressSanitizer Valgrind Gcov How do they work? How should their results be interpreted? What have they and have they not delivered for us in practice? How can we integrate them into our workflow going forward? How can we structure our code better to work with tools like these?

uk: вхід без авторизації

name: Static analysis, test coverage, and other 12+ letter words: A tour of ways to find and prevent bugs in PostgreSQL


uk: Комп'ютерні науки

description: or "How We Can Overtake Every Other Database" According to Jeff Barr, the Chief Evangelist for Amazon Web Services, "Over the past few years, PostgreSQL has become the preferred open source relational database for many enterprise developers and start-ups, powering leading geospatial and mobile applications." Through the hard work from the major contributors of our community and certain market events in the data world, it is indeed true that many new projects, both open and closed source, are defaulting to PostgreSQL as their data storage choice. So what is preventing PostgreSQL from becoming the default database for everyone? We will explore the PostgreSQL Advocacy efforts to date which has brought us to this crossroads, and how through a combination of our ongoing product development and a bit of community marketing prowess, we as a community can make "PostgreSQL" as household technology name over the next five years. According to Jeff Barr, the Chief Evangelist for Amazon Web Services, "Over the past few years, PostgreSQL has become the preferred open source relational database for many enterprise developers and start-ups, powering leading geospatial and mobile applications." Through the hard work from the major contributors of our community and certain market events in the data world, it is indeed true that many new projects, both open and closed source, are defaulting to PostgreSQL as their data storage choice. We know that PostgreSQL is an amazing product, and those who know the realm of technology agree. So what is preventing PostgreSQL from becoming the default database for everyone? In order to solve this problem, we will first explore the PostgreSQL advocacy efforts to date. We will take a look at what features have helped propel PostgreSQL into its "newfound" limelight and what the community has done to create a "buzz" around PostgreSQL to capture more users. We will then arrive at the present day crossroads, and look at different strategies to more effectively market ourselves as a product and a community, including how to: Actively make more people aware of PostgreSQL and what it can offer Make it easier for new people to adopt PostgreSQL Determine the pain points of our current users and figure out how to fix them Measure the success of our advocacy efforts Through these combined efforts on the development and advocacy fronts, we as a community can make PostgreSQL as household technology name over the next five years.

uk: вхід без авторизації

name: The Next Five Years for PostgreSQL Advocacy


The Worst Day of Your Life

uk: Комп'ютерні науки

description: Recovering from Crises and Disasters What do you do when the worst happens? It could be a catastrophic hardware (or even data center) failure, a badly placed rm -rf *, or a PostgreSQL bug. We'll discuss how to recover from disasters that are far outside the usual operating procedure... and how to avoid getting into them in the first place. Every DBA with real-life experience knows that sinking feeling when you realize that something terrible has happened: PostgreSQL crashes with a PANIC message, you realize you were on the production system when you dropped that table, or you get a status update that "us-east is currently experiencing problems." What do you do? There's no single solution to catastrophic problems, but we can talk about strategies that might help you keep a cool head while everything around you is losing theirs. We'll talk about things like: Dealing with PostgreSQL bugs. Catastrophic hardware failures. Application and operator error. And, of course, we'll discuss what you need to do in advance to make the Worst Day of Your Life a little bit less traumatic: Backup and recovery strategies and tradeoffs. Upgrade procedures. Planning for business continuity in major disasters.

uk: вхід без авторизації


Why UPSERT is weird

uk: Комп'ютерні науки

description: Counterintuitive lessons learned from the implementation effort Talk that examines implementation process on the INSERT...ON DUPLICATE KEY LOCK FOR UPDATE feature proposed for PostgreSQL. "UPSERT" can loosely be described as a DBMS feature that allows a DML statement's author to atomically either insert a row, or on the basis of the row already existing, update that existing row instead, while safely giving little to no further thought to concurrency. One of those two outcomes must be guaranteed, regardless of concurrent activity, which is the essential property of UPSERT. Examples include MySQL's INSERT...ON DUPLICATE KEY UPDATE, or the SQL standard's MERGE statement (as implemented in a number of proprietary systems). The absence of this feature from Postgres has been a long-standing complaint from Postgres users, and an effort to implement an extension to the Postgres INSERT statement, that implements something broadly in line with user expectations in this area was undertaken in 2013. This talk considers the experience of working on that implementation, and examines how the proposed INSERT...ON DUPLICATE KEY LOCK FOR UPDATE feature deals with concurrency issues, visibility issues, the interplay of value locking and row locking, and the general avoidance of unprincipled deadlocking. In order to come up with an implementation that satisfied user expectations, while still preserving and respecting long standing abstractions and conceptual precepts, some interesting and counterintuitive choices were made to resolve the fundamental tension in tying value locking (as always required on a limited scale for unique index enforcement) to row locking (to LOCK FOR UPDATE). Finally, the talk examines the strategic significance of UPSERT, and further use-cases enabled by the feature in the area of multi-master replication conflict resolution.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: Instant Mapping Applications with PostGIS and Nodejs Learn how to write PostgreSQL-backed applications that can be automatically cloned, built, monitored, and scaled in a matter of minutes. We'll walk you through all of the steps involved in developing and scaling your own Node.js-powered PostGIS mapping application on the OpenShift cloud hosting platform. By the end of this talk, you should be able to instantly deploy your own PostgreSQL-backed network applications to your own public or private cloud in a single command. This talk includes a brief architectural overview of OpenShift, a freely-available Open Source hosting platform. Folks who want to follow along should bring their own laptop with git, rubygems, and nodejs installed & available.

uk: вхід без авторизації

name: Postgis in the Open Cloud: Writing portable open Source mapping applications with LEAFLET and OPENSHIFT


Schema-less data in PostgreSQL

keywords:
Alexander Korotkov
Oleg Bartunov
Teodor Sigaev

uk: Комп'ютерні науки

description: An efficient indexing of nested structures We present a prototype of new access method, heavily based on GIN and optimized for efficient indexing of nested structures like hstore and json(b). Introducing of the nested hstore and jsonb in PostgreSQL brought new challenge to the developers, namely, an efficient indexing of hierarchical keys. Those keys are consist of duplicated strings, which made index to be uselessly huge if store key-value pairs independently. We propose to replace btree data structure, which used in GIN to index keys, by digital tree. To do this in 'right way', we would like to experiment with hybrid access method based on of SP-GiST and GIN. This is a first step in making GIN more flexible to support richer set of queries. In principle, one could be able to use other than btree data structure to index not just keys, but also the posting lists.

uk: вхід без авторизації

name: PGCon - PostgreSQL Conference for Users and Developers, Andrea Ross


GIN in 9.4 and further

keywords:
Alexander Korotkov
Oleg Bartunov

uk: Комп'ютерні науки

description: This talk presents set of GIN advances in PostgreSQL 9.4 and further which brings GIN to new level of performance and extendability. Most important advances are: posting lists compression, fast-scan algorithm, storing additional information and index-based ranking. This talk presents set of GIN advances: Compression posting lists. Indexes become 2 times smaller without any work in opclass. pg_upgrade is supported, old indexes will be recompressed on the fly. Fast scan algorithm. Fast scan allows GIN to skip parts of large posting trees during index scan. It dramatically improve performance of hstore and json search operators as well as FTS "frequentterm & rareterm" case. In order to use this improvement three-state logic support required in "consistent" opclass method. Storing additional (opclass defined) information in posting lists. Usage of additional information for filtering enables new features for GIN opclasses: better phrase search, better array similarity search, inverse FTS search (search for tsqueries matching tsvector), inverse regex search (search for regexes matching string), better string similarity using positioned n-grams. Index based ranking. This improvement allows GIN to return results in opclass defined manner. Most important application is returning results in relevance order for FTS which dramatically reduces IO load. But there are other applications like returns arrays in similarity order. We present the results of benchmarks for FTS using several datasets (6 M and 15 M documents) and real-life load for PostgreSQL and Sphinx full-text search engines and demonstrate that improved PostgreSQL FTS (with all ACID overhead) outperforms the standalone Sphinx search engine.

uk: вхід без авторизації


XML, HSTORE, JSON, JSONB - OH MY!

uk: Комп'ютерні науки

description: Use the unstructured data type that's right for you. PostgreSQL 9.4 adds a new unstructured data type, JSONB, a variant of JSON optimized for storing objects. JSONB complements the existing XML and JSON document objects, as well as HSTORE. Which one is right for you? We'll take a tour of the features of each, and their advantages and disadvantages for various use cases, all illustrated with real-world examples. There has been a lot of work on the representation of unstructured data in PostgreSQL, culminating in the addition of the JSONB type in the forthcoming 9.4 release. JSONB complements the existing HSTORE, XML, and JSON types, not to mention arrays. With so many options, which do you use? As usual it depends on your use case. In this presentation, we'll review the unstructured data types in PostgreSQL, and look at their advantages and disadvantages for: Document storage Configuration management A "schemaless database" Object serialization Entity/Attribute/Value models Path queries

uk: вхід без авторизації


uk: Комп'ютерні науки

description: New features by demo This all-demo, no-slide talk will show off 9.5's new features. With every new Postgres release comes new features and improvements to make your life easier. Come see some of the new 9.5 features in action and learn how this next release will make your life better.

uk: вхід без авторизації

name: 9.5 Coming to You Live: Upcoming PostgreSQL 9.5 Features


All the Dirt on VACUUM

uk: Комп'ютерні науки

description: The use of Multi-Version Concurrency Control (MVCC) is perhaps one of the most powerful features PostgreSQL has to offer, but it can be a source of confusion for new and experienced users alike. In this talk we will provide an in-depth walkthrough of why Postgres needs to vacuum and what vacuum does. Topics: - MVCC details - HOT overview - Identifying tuples to be vacuumed/frozen - VACUUM and indexes - Vacuuming heap pages

uk: вхід без авторизації


uk: Комп'ютерні науки

uk: вхід без авторизації

name: Data Encoding Schemes: Scales & Measurements


uk: Комп'ютерні науки

description: BiDirectional Replication In the course of the BDR (BiDirectional Replication) project we have worked on delivering robust, feature-full and fast asynchronous multi-master replication for postgres. In addition we have started the UDR project, sharing most of the code and infrastructure with BDR, which provides unidirectional logical replication for the many cases where multi-master replication is not required. To implement BDR a lot of features have already been integrated into core PostgreSQL (9.4). Now that 9.4 is released and BDR/UDR is in production in several complex environment there's some important discussions to be had about what can and what cannot be integrated into core PostgreSQL. We will discuss: Which features are in core postgres Which features does BDR/UDR provide on top of that What can be integrated into core PostgreSQL and how Future features Problems found during the development

uk: вхід без авторизації

name: Future(s) of PostgreSQL (Multi-Master) Replication


GSoC2014 - Sharing Code and Experience

uk: Комп'ютерні науки

description: This presentation is about my experience in the FOSS world and contributing to PostgreSQL as a Google Summer of Code 2014 student. In this presentation I'll talk about all my involvement with the FOSS world and how it change my life and career in many ways. I'll explain how Google Summer of Code works and the importance of this program to the open-source communities. Some points covered: - who can apply - how to apply - how you can help the PostgreSQL community - principal events

uk: вхід без авторизації


Heavy Duty Backup with PgBackRest

uk: Комп'ютерні науки

description: PgBackRest is a backup system developed at Resonate and open sourced to address issues around the backup of databases that measure in tens of terabytes. It supports per file checksums, compression, partial/failed backup resume, high-performance parallel transfer, async archiving, tablespaces, expiration, full/differential/incremental, local/remote operation via SSH, hard-linking, and more. PgBackRest is written in Perl and does not depend on rsync or tar but instead performs its own deltas which gives it maximum flexibility. This talk will introduce the features, give sample configurations, and talk about design philosophy. PgBackRest aims to be a simple backup and restore system that can seamlessly scale up to the largest databases and workloads. Instead of relying on traditional backup tools like tar and rsync, PgBackRest implements all backup features internally and features a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security. Each thread requires only one SSH connection for remote backups. Primary PgBackRest features: Local or remote backup Multi-threaded backup/restore for performance Checksums Safe backups (checks that logs required for consistency are present before backup completes) Full, differential, and incremental backups Backup rotation (and minimum retention rules with optional separate retention for archive) In-stream compression/decompression Archiving and retrieval of logs for replicas/restores built in Async archiving for very busy systems (including space limits) Backup directories are consistent Postgres clusters (when hardlinks are on and compression is off) Tablespace support Restore delta option Restore using timestamp/size or checksum Restore remapping base/tablespaces

uk: вхід без авторизації


uk: Комп'ютерні науки

description: Why, when and how you can integrate documents and key-value pairs into your relational model There is a pitched battle going on between the relational, document-based, key-value and other data models. PostgreSQL is uniquely capable of leveraging many of the strengths of multiple data models with JSON(b), HSTORE, XML, ltree data types, arrays and related functions. This presentation outlines the use-cases, benefits and limitations of document-based, key-value and hierarchical data models. It then presents practical advice and code snippets for incorporating them into PostgreSQL's relational framework. The presentation ends with SQL examples and code snippets for loading, accessing and modifying (where possible) JSON, HSTORE, XML, ltree and array data types. This presentation begins with a very quick review of the rationale, benefits and implications of the relational data model. It then does the same for document-based models and hierarchical models. The balance of the presentation works with three publicly available data sets, world-wide airports, Wikipedia Inbox key-value pairs and Google address JSON objects, showing how they can be be incorporated into a simple relational model. The presentation also includes snippets of code for loading the files and accessing elements. The full SQL, and shell code will be available on the web site.

uk: вхід без авторизації

name: If you can't Beat 'em, Join 'em!: Integration NoSQL data elements into a relational model


uk: Комп'ютерні науки

description: PostgreSQL Enterprise Consortium (PGECons for short) is an organization that consists of major IT companies in Japan, aiming to promote PostgreSQL to enterprise users in the country. Since 2012 when PGECons was established, we have been doing surveys of PostgreSQL's functions and performance to PGECons members to estimate how PostgreSQL well meets their requirements. In this talk, we will focus on some of major requests from the surveys, including enhancement of table partitioning and error messages handling. From the enterprise users' point of view, we would like to share these obstacles behind the requests that might limit PostgreSQL's acceptance, in order to cope with these issues with the community. [Partitioning] More the size of a database grows, the more difficult the design and operation of the system become. For example, a sales accounting application has such difficulty, which is often solved by using horizontal table partitioning. We have surveyed how PostgreSQL can be applied to such kind of applications to the member of PGECons, and evaluated performance of table partitioning. These surveys result suggested PostgreSQL partitioning issues below. larger number of partitions slows query response time definition and usage of partitioned table are intricate From additional surveys about proprietary DBMS usage to the members, we will show what are needed in partitioning and the features of the application areas that are important but difficult to be supported by PostgreSQL. Then, we will point out that enhancing partitioning functions will make PostgreSQL to be spread to the aforementioned areas. [Error messages] When shooting a trouble about PostgreSQL, a user analyzes a trouble to identify the cause based on the error logs and error messages. However, when we analyze a trouble with only SQLSTATE code we sometimes cannot identify the exact cause because an error code is often assigned to multiple different error messages. Also, we have similar difficulty when analyzing a trouble from error messages. To cope with these problems, Fujitsu Limited, a member of PGECons, delivers customers a list of error messages sorted by SQLSTATE codes to search the corrective actions. In the list, an identifier is added to each item so that customers immediately decide corrective actions and communicate smoothly with support staffs. Based on our activities above, we will propose error message systems which will bring following two merits: accelerate decisions on corrective actions enhance the self support by users In this presentation, we will show the needs and actions from members of PGECons as enterprise users, to share the way of improvement, which should help promote PostgreSQL among the community.

uk: вхід без авторизації

name: Improving PostgreSQL for Greater Enterprise Adoption


Inside PostgreSQL Shared Memory

uk: Комп'ютерні науки

description: This talk is for people who want to understand how PostgreSQL shares information among processes using shared memory. The talk covers the internal data page format, usage of the shared buffers, locking methods, and various other shared memory data structures.

uk: вхід без авторизації


...Lag: What's wrong with my slave?

uk: Комп'ютерні науки

description: Most of the time, a streaming replication slave in the same data center is so close to the master that lag can be measured in milliseconds. However when it's not, that lag can be baffling at best, and catastrophic at worst. We will look at all things lag; strategies of monitoring, configuration options to fit application needs, diagnosing common issues and real cases of 'what went wrong'. If you google from "postgres streaming replication lag" (go ahead, I'll wait...) your result set will include much information on set up and monitoring, but very little on diagnosing and even less on correcting. This talk is an attempt to fill that gap. We will start with the basics of monitoring and trending over time, look at configuration options and 'gotchas' for making your slaves trusted read sources, diagnose hardware and system factors, and finally share the pain of elusive lag patterns that took days, if not weeks to figure out. This talk takes a broad look at system health. Many factors contribute to making a database cluster run perfectly; disk speed, network latency, user query patterns, etc., etc. It can be easy to over look, or take for granted things that may strongly effect how close a slave follows the master. In fall of 2014 iParadigms converted 8 server clusters across two data centers to streaming replication, allowing us to find and document many such issues.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: Keeping track of changes in your database schema can be challenging. In this talk I will discuss the advantages of using Flyway to effectively manage this issue. Migrations are an essential tool for both developers and administrators. Developers can quickly recreate a database from scratch and incrementally modify their development database along with their code and tests. Similarly, administrators can determine the current state of any database and easily migrate to a newer one. Most importantly, schema and data changes can be thoroughly reviewed and tested before going to production. In this talk, I will discuss the benefits of using Flyway to manage migrations. Specifically, I will: - Show why migrations are useful - Introduce Flyway and how to use it - Focus on using Flyway from the command line using migrations written in sql - Help you determine which changes should be in your migrations - Discuss how to create a base migration from your existing database - Cover strategies for dealing with global objects - Show how to integrate Flyway with Jenkins

uk: вхід без авторизації

name: Managing your schema: Using migrations for consistency, repeatability, and sanity


uk: Комп'ютерні науки

description: A lot has changed since SQL:92 SQL has gone out of fashion lately --- partly due to the NoSQL movement, but mostly because SQL is often still used like 20 years ago. As a matter of fact, the SQL standard continued to evolve during the past decades resulting in the current release of 2011. In this session, we will go through the most important additions since the widely known SQL-92, explain how they work and how PostgreSQL extends them. We will cover common table expressions and window functions in detail and have a very short look at the temporal features of SQL:2011 and the related features of PostgreSQL.

uk: вхід без авторизації

name:
Modern SQL in PostgreSQL: Still using Windows 3.1?
So why stick to SQL-92?


uk: Комп'ютерні науки

description: pg_statsinfo comes with new features There are many information vanishes as the server operates. pgstatsinfo/pgstatsreporter is a monitoring tool which records such various status and statistics of PostgreSQL server and lets you see them in graphical and interactive way. It is very usable not only for DBAs to check the health of the server daily, but for technical support to find out what happened in the past on the remote site. It is widely adopted among our systems using PostgreSQL and the new pgstatsinfo 3.1 has new features to support them more. The new pg_statsinfo 3.1 has the following new features in comparison to 2.5. This talk will introduce these features with demo and dig inside some of them. Collecting plan statistics. Plan statistics is based on an original pgstatstatements-like extension named pgstoreplans, which is a similar tool to 2ndQuadrant's pgstatplans but it still differs in some points to fit to pg_stastinfo. Statistics of autovacuum/analyze including cancellation stats. Cancellation stats would be in some situations. Storing server logs into the repositiry database. Stored logs are examined using filtering feature of pgstatsreporter. Storing alerts previously only emitted into server log. Alerts gets more valuable with pgstatsreporter.

uk: вхід без авторизації

name: Monitor more of PostgreSQL: Introduction of New Features of pg_statsinfo


Multi-Tenancy in PostgreSQL

uk: Комп'ютерні науки

description: This talk is about the need of multi-tenancy in PostgreSQL, and the way to achieve multi-tenancy in PostgreSQL. What is a multi-tenant cluster? Why multi-tenant cluster is needed? PostgreSQL Provides multi-tenancy with the following - Shared Database, Shared Namespace - Separate Databases - Shared Database, Separate Namespace However multi-tenancy means more than this. - Issues with a multi-tenant cluster - What can be done and what can we do to make it easier. This talk will propose a multi tenanted architecture for PostgreSQL, to make it the database of choice in a cloud environment. Multi tenanted architecture is one of the key requirements for any software to be efficiently deployed in the cloud. As more and more databases are made available 'as-a-Service' in cloud offerings, it is necessary to take stock of the features in PostgreSQL to analyse how cloud friendly they are, especially for a multi-tenanted infrastructure. This talk will mainly focus on what functionalities are needed in PostgreSQL to make it truly cloud friendly. PostgreSQL needs to have the functionalities that will make it the database of choice for service providers in the cloud. This can be achieved within the current architecture of PostgreSQL by developing new features that will satisfy these requirements.

uk: вхід без авторизації


Parallel Sequential Scan

uk: Комп'ютерні науки

description: Unleashing a heard of elephants Parallel query is close to becoming a reality in PostgreSQL! A year ago, much of the low-level infrastructure needed for parallelism, such as dynamic shared memory and dynamic background workers, had been completed, but no user-visible facilities made use of this infrastructure. Major work on error handling, transaction control, and state sharing has been completed, and further patches, including a patch for parallel sequential scan, are pending. In this talk, we will talk about parallel sequential scan itself, including performance considerations, the work allocation strategy, and the cost model; and we will also discuss the infrastructure that supports parallel sequential scan, including state sharing for GUCs, transaction state, snapshots, and combo CIDs; error handling and transaction management; and the handling of heavyweight locking. Finally, we'll discuss the future of parallelism in PostgreSQL now that the basic infrastructure is (mostly) complete.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: PostgreSQL extension to scale out real-time reads and writes pg_shard is an open source sharding extension for PostgreSQL. It shards PostgreSQL tables for horizontal scale, and replicates them for high availability. The extension also seamlessly distributes SQL statements, without requiring any changes to the application layer. pg_shard addresses many NoSQL use-cases, and becomes more powerful with the new JSONB data type. Further, the extension leverages the rich analytic capabilities in PostgreSQL, and enables real-time analytics for big data sets. In this talk, we first summarize challenges in distributed systems associated with scaling out databases. We then describe "logical sharding", and discuss how it helps overcome these challenges. Next, we show how pg_shard uses hook APIs, such as the planner and executor hooks, to make PostgreSQL a powerful distributed database. We then cover example customer use-cases, and conclude with a futuristic demo: a distributed table with JSONB fields, backed by a dynamically changing row and columnar store. pg_shard is an open source sharding extension for PostgreSQL. It shards PostgreSQL tables for horizontal scale, and replicates them for high availability. The extension also seamlessly distributes SQL statements, without requiring any changes to the application layer. pg_shard addresses many NoSQL use-cases, and becomes more powerful with the new JSONB data type. Further, the extension leverages the rich analytic capabilities in PostgreSQL, and enables real-time analytics for big data sets. In this talk, we first summarize challenges in distributed systems: dynamically scaling a cluster when new machines are added or old ones fail, and distributed consistency semantics in the face of failures. We then describe "logical sharding", and show how it helps overcome these challenges. We also discuss this idea's application to Postgres. Next, we show how pg_shard uses hook APIs, such as the planner and executor hooks, to make PostgreSQL a powerful distributed database. We then cover example customer use-cases, and conclude with a futuristic demo: a distributed table with JSONB fields, backed by a dynamically changing row and columnar store.

uk: вхід без авторизації

name: pg_shard: Shard and Scale Out PostgreSQL


Rethinking JSONB

uk: Комп'ютерні науки

description: PostgreSQL 9.4 has introduced JSONB, a structured format for storing JSON, which provides many users with the new opportunity: an effective storing and querying JSON documents inside ACID relational database. While users have notice a great jsonb performance, their feedback also reveals some hidden problems with current jsonb implementation. We want to discuss different approaches to resolve aforementioned problems and present several proof-of-conceps, so we could rethink jsonb for 9.6.

uk: вхід без авторизації


Row Level Security

uk: Комп'ютерні науки

description: In this talk we'll review Row-Level Security (RLS), provide examples and use-cases, discuss the work which has been done on adding Row Level Security to PostgreSQL and the current state of that effort. PostgreSQL has long had a complex and interesting set of permissions available through the GRANT system. There is another system which exists in many other RDBMS's known as row-level security (RLS), where the rows returned is filtered based on a policy implemented on the table.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: This paper will main talk about the scalability and performance improvements done in PostgreSQL 9.5 and will discuss about the improvements that can be done to improve the scalability for both Write and Read operations. The paper will focus on pain points of Buffer Management in PostgreSQL and the improvements done in 9.5 to improve the situation along with performance data. It will also describe in brief the performance improvements done in 9.5. It will also discuss the locking bottlenecks due to various locks (lightweight locks and spinlocks) taken during Read operation and what could be done to further scale the Read operation. The other part of the paper focusses on improving the Write-workload in PostgreSQL. In this part we will discuss about the frequency of writes done by backend operations (along with data) due to limitations of current bgwriter algorithm and some ideas to improve the performance by reducing writes done by backend. It will also discuss about the concurrency bottlenecks in write operation and some ideas to mitigate the same.

uk: вхід без авторизації

name: Scalability and Performance Improvements in PostgreSQL 9.5


uk: Комп'ютерні науки

description: In Current MVCC solution of PG, ProcArrayLock is the major bottleneck on many core machine 120+ and can scale up to 30 connections in TPCC test. Done experiment with lock free MVCC solution, and it can Scale up to 120 cores. In Current MVCC solution of PG, ProcArrayLock is the major bottleneck on many core machine 120+ and can scale up to 30 connections in TPCC test. Done experiment with lock free MVCC solution, and it can Scale up to 120 cores. We have taken the CSN based solution proposed in PG community, and implemented a lock free version of the same. By considering the High Memory and other resources in many core machines, locks are avoided in all the performance patch and only in some rare paths locks are used.

uk: вхід без авторизації

name: Scalable MVCC solution for Many core machines


SchemaVersus: Space battles in Postgres

uk: Комп'ютерні науки

description: 1v1 Schemaverse Battles A tournament of 1v1 schemaverse battles, each round taking only about 10 minutes. No prepared scripts allowed! The Schemaverse is a space-based strategy game implemented entirely within a PostgreSQL database. Compete against other players using raw SQL commands to command your fleet. Or, if your PL/pgSQL-foo is strong, wield it to write AI and have your fleet command itself! This year, rather than the classic large space battle, the rounds will be 1v1 and only take 10 minutes each. There will also be NO pre-created scripts allowed. Matches will be broadcast live for all to see and a ladder updated after each round.

uk: вхід без авторизації


Shabang

uk: Комп'ютерні науки

description: Scripting with Postgres Sometimes bash is just the way to go! This talk will cover tips and techniques for effective bash scripting with PostgreSQL. Sometimes bash is just the way to go! This talk will cover tips and techniques for effective bash scripting with PostgreSQL. It will include guidance about: Pros/cons of shell scripts Function library creation and use Executing SQL Set/get PostgreSQL data from/into script variables Keeping PostgreSQL functions in sync with scripts Locking Doing work in parallel Ensuring cleanup This is a source-code heavy talk. Moderate experience with both bash scripting and PostgreSQL is needed to get the most out of it.

uk: вхід без авторизації


Shootout at the PAAS Corral

uk: Комп'ютерні науки

description: Shootout at the PAAS Corral head-to-head for PostgreSQL cloud platforms Where should you run your PostgreSQL in the cloud? Join us for a comparison of features, pricing and performance between various cloud options, including most or all of EC2, Amazon RDS, Heroku, OpenShift, Google Compute, and the Rackspace Cloud. To determine which cloud is the fastest, cheapest and best, over the next few months Josh Berkus and others will be running a series of performance benchmarks against several of the many cloud hosting options available for PostgreSQL. This will include most or all of EC2, Amazon RDS, Heroku, OpenShift, Google Compute, and the Rackspace Cloud. The results will be presented to you in this talk, including: Benchmarking methdology Cost comparison for each configuration Feature differences Performance scores

uk: вхід без авторизації


The Art of Performance Evaluation

uk: Комп'ютерні науки

description: "Contrary to common belief, performance evaluation is an art." (Raj Jain, 1991) Successful performance evaluation may not be achieved with merely executing common benchmarking tools. This talk presents fundamental principles of performance evaluation and how you can put them into practice. Do you understand what exactly "pgbench" does? Is it appropriate workload for your performance evaluation goal? Common benchmarking tools like "pgbench" are handy for just comparing system A and system B, but if you intend to deeply understand the performance of your system, answers to these questions are critical. In order to conduct a meaningful performance evaluation, the methodology should be elegantly designed to meet the goal of the evaluation: choose metrics for the goal, and choose observation techniques for the metrics. Each step requires careful consideration and deep knowledge about the target system. It cannot be done mechanically. This is why performance evaluation is an art. This talk presents principles of designing performance evaluations and shows how you can put them into practice by introducing the speaker's experiences of performance evaluations with PostgreSQL.

uk: вхід без авторизації


Tracing PostgreSQL performance

uk: Комп'ютерні науки

uk: вхід без авторизації


uk: Комп'ютерні науки

description: Managing transactions involving multiple foreign servers. PostgreSQL has Foreign Data Wrappers and they are writable too! Upcoming features like partitioning, foreign table inheritance and join-push down for foreign tables pave the path for sharding. One missing piece in the puzzle is the distributed transaction manager required to maintain the atomicity and consistency of transactions involving foreign servers. The presentation talks about the current status of such transactions and discusses the path forward towards distributed transaction manager. Support for writable foreign tables was added in PostgreSQL 9.3. As of now, atomicity and consistency is guaranteed, when a transaction makes changes to at most a single foreign server. It fails to do so when changes are made to multiple foreign servers. In order to achieve atomicity and consistency of transactions involving multiple foreign servers, PostgreSQL needs to take up the role of a distributed transaction manager. The talk covers the current status of distributed transactions. It further explores protocol to drive distributed transactions and infrastructure necessary to overcome various hardware, software and network failures during a distributed transaction. It also covers the use cases like data federation and sharding.

uk: вхід без авторизації

name: Transacting with foreign servers: Two's company, three is ...


uk: Комп'ютерні науки

description: Postgres 9.4 introduced the new jsonb type, However, it is missing some functions, particularly for json composition, that are needed by many users. In this talk we present an extension that provides some of these functions, and work to incorporate the functions in 9.5. Operations include replacement and deletion of array elements and object fields, and composition by concatentation of objects and arrays. Arrays can also be concatenated with scalar values, and array elements can be replaced or deleted by counting from either end of the array. Thus we have the ability to use json arrays as queues and stacks, with basic push/pop and shift/unshift capability. A pretty print function for jsonb is also provided. We will also outline what work we think remains, and discuss possible ideas on how to make json composition more naturally expressed.

uk: вхід без авторизації

name: Update and Delete operations for jsonb (part 1 of 2): Providing some needed functions and operators for jsonb


uk: Комп'ютерні науки

description: providing some needed functions and operators for jsonb Postgres 9.4 introduced the new jsonb type, However, it is missing some functions, particularly for json composition, that are needed by many users. In this talk we present an extension that provides some of these functions, and work to incorporate the functions in 9.5. Operations include replacement and deletion of array elements and object fields, and composition by concatentation of objects and arrays. Arrays can also be concatenated with scalar values, and array elements can be replaced or deleted by counting from either end of the array. Thus we have the ability to use json arrays as queues and stacks, with basic push/pop and shift/unshift capability. A pretty print function for jsonb is also provided. We will also outline what work we think remains, and discuss possible ideas on how to make json composition more naturally expressed.

uk: вхід без авторизації

name: Update and Delete operations for jsonb (part 2 of 2): New Features and Directions for jsonb in PostgreSQL


Warm standby done right

uk: Комп'ютерні науки

description: People has been setting up warm standby systems with streaming replication since version 9.0, and even longer with file-based log-shipping. However, there has been a few pitfalls that many people don't know about, while others have simply accepted the risks. PostgreSQL 9.5 brings a bunch of new features and subtle changes that make warm standby setups more robust than ever. In 9.5, the interaction between a WAL archive and failover has been revised. pg_rewind makes it possible to resynchronize an old master server after failover - even an unplanned one. Replication slots, already introduced in 9.4, make the behaviour of a standby falling behind nicer. This presentation explains the changes, and why they were needed. Finally, I'm going to walk through setting up a simple, robust, two server hot standby system, using only built-in tools and simple shell scripts, taking advantage of the new features.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: State of the different way of managing binaries packages for FreeBSD from building to installing/upgrading your servers/jails This talk will provide an overview of how to do modern package management with FreeBSD. From building farms, QA Validation, hosting and deployemnt, new features of pkg(8) 1.1. New way of deploying FreeBSD: packaged base bootstrapped via pkg(8). New way of deploying/managing FreeBSD jails: packaged world, bootstrapped/installed via pkg(8)

uk: вхід без авторизації

name: Modern package management: Building, deploying, installing, upgrading packages on FreeBSD

name: Berkeley System Distribution (BSD), Andrea Ross


uk: Комп'ютерні науки

description: Maintaining the port of Firefox to OpenBSD is an interesting challenge, since the web & its related technologies are evolving faster and faster, and third-party OS are struggling more and more to keep up the pace with a fast-moving target such as Mozilla. I'll explain how i got caught into this by accident in the beginning of 2010, what are the key things to know about Mozilla development when coming from another big OSS project, how to properly cooperate with upstream, and how i managed to wrap up a workflow that eases the port updates at each new Mozilla release. Firefox (and thunderbird !) have been ported to OpenBSD around 2004, and since then there's been a constant work to keep them working fine, and up-to-date. Especially since the switch to a fast release schedule, you need to track very closely what happens upstream to ensure nothing breaks on third party operating systems at each new release, since Mozilla only considers Windows, Linux and MacOSX as tier-1 platforms. More and more code (often depending on os-specific code) is dumped into mozilla's source tree, and you also need to make sure that code also work on your os, or then has to be made optional. You also need to take special care to push back your fixes upstream for two reasons : making maintainance a breeze over time, and letting upstream know that you exist and that there are quite some users on other platforms. The developments in the next web APIs are making it crucial to have a fully functional browser on your favorite OS, so we'll also see what's needed for that.

uk: вхід без авторизації

name: Mozilla on OpenBSD: When the lizard meets puffy


uk: Комп'ютерні науки

description: Bambi Meets Godzilla: They Elope The world of open source and the world of commercial software intersect in profound and sometimes surprising ways. In some sense the two are like oil and water, but in other ways they can build on each other. This talk describes some of those differences — and ways they can enhance one another — using the sendmail open source mail transfer agent and Sendmail, Inc. as a case study.

uk: вхід без авторизації

name: Open Source Meets the Commercial World: The opening session


uk: Комп'ютерні науки

description: Busy http servers, storage appliances and toasters are some notorious success deployments of NetBSD. Unfortunately not everybody is aware of some other gaps that NetBSD has filled. Once upon a time, there was a brave small team of programmers who did dare to choose NetBSD as a platform for their R&D software engineering department inside an aerospace and defense company. This resulted in some unusual use cases for NetBSD. This talk will present (1) a NetBSD-OS based tracking radar; (2) how NetBSD-OS served as a platform for supporting the software engineering department to design and implement software systems for rocket tracking radars; (3) how NetBSD Project served as a knowledge base for modelling that department's engineering processes. Furthermore, this talk will show how the unique features of NetBSD made it the most suitable OS for building rocket launching tracking radars deployed up to 3000 Km from the software engineering R&D lab, and how it helped to address challenges such as: Real-time data acquisition and processing Advanced visualization of radar data Radar systems simulation Air defense systems simulation Radar systems tests and quality assurance We'll also show how the choice for NetBSD made by an young software engineer, became a fundamental resource to deal with several aspects of bootstrapping an R&D software engineering department from ground up. Beyond an open-source OS, the NetBSD Project has an inherent open-governance and open-engineering nature, which made it a "body of knowledge" repository we used to find references for several software engineering related subjects.

uk: вхід без авторизації

name: A NetBSD-based Radar in a Rocket Launching Center: The untold story of how NetBSD Project served as an engineering platform for a small rocket launching tracking systems R&D company.


uk: Комп'ютерні науки

description: The FreeBSD security model has been developed over thirty years of evolving consumer needs. Many of the key developments have come from the contributions of an active security research community. This talk describes the underlying model and its practical implementation, from its origins in the UNIX process model and file permissions, to more recent additions: the Capsicum capability model, lightweight Jail virtualization, Mandatory Access Control, and security event auditing. These elements combine to meet the requirements of diverse systems ranging across hand-held computing devices, network devices, storage appliances, and Internet service-provider's large-scale hosting environments.

uk: вхід без авторизації

name: An Overview of Security in the FreeBSD Kernel


uk: Комп'ютерні науки

description: In PC-BSD 9.x every installation is fully-scripted, due to the the pc-sysinstall backend. This backend can also be used to quickly automate the deployment of FreeBSD servers and PC-BSD desktops using a PXE boot environment. In PC-BSD & TrueOS

uk: вхід без авторизації

name: Automating the deployment of FreeBSD & PC-BSD systems: System deployment using PXE and pc-sysinstall


uk: Комп'ютерні науки

description: System optimization and tuning is tricky business. Benchmarking such systems is even more complicated as the number of things which can go wrong at doubles at the least. FreeBSD is best known for its killer features, but it also includes over 2500 sysctls, most of which can be tuned to do something interesting. This talk aims to give an overview of some of the more interesting things which can be tuned in FreeBSD, and advice on how to avoid the most common errors in benchmarking FreeBSD. Tuning a system heavily depends on hardware present in the system, so all tuning advice will necessary contain system-specific parts, but overall there is much to discuss when talking about optimizing specific systems: networking, storage, file systems, even the CPU scheduler. Networking is still very much dependant on the quality of the NIC and its driver, but large parts of the systems such as ZFS are pure software and can benefit from tweaks and tuning which slightly alter the behaviour of algorithms. Things get very complicated when comparing different hardware configurations, and even more when comparing different operating systems. Doing a good benchmark of two unrelated operating systems is tricky because it requires similar tune-ups to both systems. This talk will try to explain where the pitfalls are, and also present some field results.

uk: вхід без авторизації

name: Benchmarking FreeBSD: Benchmarking - what not to do and how to avoid it if possible


uk: Комп'ютерні науки

description: After focusing on improving NetBSD support for tablet hardware, NetBSD enthusiast and DeforaOS developer Pierre Pronchery (khorben@) is working on running NetBSD on smartphone hardware, with the OMAP-based Nokia N900 as the main target. After a quick summary of the different steps taken, he will introduce the user interface, installation procedure and underlying specificities to this setup. If all goes well, he will even be able to give a call! NetBSD is a very portable Operating System, which supports a plethora of hardware devices based on an ARM CPU and SoC (System on a Chip). Among them is the OMAP family from Texas Instruments; popularized by the BeagleBoard demonstration board, it is well supported by a number of Open Source systems. As it happens, there is at least one smartphone built around the same platform, namely the N900 from Nokia. Being well supported by Linux already, it is an ideal candidate for introducing modern telephony support to NetBSD. The user interface used as part of this effort originates from the DeforaOS project. Its graphical environment is based on the Gtk+ visual toolkit, and has already been declined for several embedded designs, including a phone: the Openmoko Freerunner. Introduced within pkgsrc as of the 2012Q4 release, it will hopefully provide a functional telephony environment on a NetBSD-based phone by the time of the presentation.

uk: вхід без авторизації

name: Call your NetBSD: Running NetBSD on phone hardware


uk: Комп'ютерні науки

description: Unicage software development method "Tukubai" is a comprehensive development framework for the enterprise systems (sales accounting system, payroll accounting system, corporate system, CRM system, merchandising system, enterprise system self-manufacture etc) including from development philosophy to development method, tools, coding, documentation and its business model. In 2012, rapidly growing company Universal Shell Programming Laboratory, the founder of "Tukubai" method, found that FreeBSD is better choice for them. I have been working for them as a FreeBSD consultant. In light of my own experience, I'll cover: o FreeBSD situation of the corporate activity o What's the Unicage software development method "Tukubai" o How many/Which companies uses Tukubai method o What's the FreeBSD specialized feature ush and BubunFS o How to scale the FreeBSD based Tukubai system o How to build the FreeBSD based Big data processing appliance In the middle of 2012, I developed a FreeBSD based Big data processing appliance, I'll cover that story, too.

uk: вхід без авторизації

name: FreeBSD based Japanese Enterprise System and Tukubai Method: Unicage software development method and scalable FreeBSD based Big data appliance


uk: Комп'ютерні науки

description: In todays IT workplace, managing and tracking servers is becoming more and more important. Many Sys Admins are responsible to not only their Boss, but to the users that want the system to run their apps and the CFO who wants to be able to depreciate them properly. We will talk about effective ways to track and manage the servers to keep everyone happy. Cover best practices with tracking and building the server including - Asset tracking - Scripted Install over PXE - Serial Console for out of band management - Package management/updating with Poudriere and pkgng - Including custom nob tuning - Configuration management using tools like - Puppet - Chef - CFEngine - Server patching with FreeBSD-Update and failover with CARP - Server retirement, i.e. Data destruction and Asset depreciation

uk: вхід без авторизації

name: FreeBSD Birth to Death: Managing the Lifecycle of a FreeBSD Server


Introduction to pkgsrc

uk: Комп'ютерні науки

description: Introduction to the Pkgsrc third-party software packaging framework. This lecture will introduce the significant differences and interesting features offered by Pkgsrc. This is the portable package system maintained by NetBSD and used also for DragonFly, SmartOS, MINIX, and various other operating systems.

uk: вхід без авторизації


uk: Комп'ютерні науки

description: High speed network communication is challenging on bare metal, and even more so in virtual machines. There we have to deal with expensive I/O instruction emulation, format manipulation, and handing off data through multiple threads, device drivers and virtual switches. Common solutions to the problem rely on hardware support (such as PCI passthrough) to make portions of the NIC directly accessible to the guest operating system, or specialized drivers (virtio-net, vmxnet, xenfront) built around a device model that is easier to emulate. These solutions can reach 10 Gbit/s and higher speeds (with suitably large frames), one order of magnitude faster than emulated conventional NICs (e.g. Intel e1000). Despite popular belief, NIC emulation is not inherently slow. In this paper we will show how we achieved VM-to-VM throughputs of 4 Mpps and latencies as low as 100us with only minimal modifications to an e1000 device driver and frontend running on KVM. Our work relies on four main components, which can be applied independently: 1) proper emulation of certain NIC features, such as interrupt mitigation, which greatly contribute to reduce the emulation overhead; 2) modified device drivers that reduce the number of I/O instructions, much more expensive on virtual machines than on real hardware; 3) a small extension of the device model, which permits shared-memory communication with the hypervisor without requiring a completely new device driver 4) a fast network backend (VALE), based on the netmap framework, which can sustain multiple millions of packets per second; With the combination of these techniques, our VM-to-VM throughput (two FreeBSD guests running on top of QEMU-KVM) went from 80 Kpps to almost 1 Mpps using socket based applications, and 4 Mpps with netmap clients running on the guest. Similarly, latency was reduced by more than 5 times, reaching values of less than 100 us. It is important that these techniques can be applied independently depending on the circumstances. In particular, #1 and #4 modify the hypervisor but do not require any change in the guest operating system. #2 introduces a minuscule change in the guest device driver, but does not touch the hypervisor. #4 relies on device driver and hypervisor changes, but these are limited to a few hundreds of lines of code, compared to the 3-5 Klines that are necessary to implement a new device driver and its corresponding frontend on the hypervisor.

uk: вхід без авторизації

name: Lightning fast networking in your virtual machine


uk: Комп'ютерні науки

description: Detailed discussion of ScaleEngine's production implementation of puppet on FreeBSD to manage many heterogeneous servers across the globe, with 70+ servers at 26 data centres in 10 countries deployed in a number of different roles (Web Hosting Cluster, HTTP Accelerator, HTTP CDN, Live Video, On-Demand Video, GSLB DNS) our needs cover a large swath of the capabilities of any management system. It is common for sysadmins to jump straight to cloud providers if immediate scale is required. This unnecessarily reduces autonomy and choice, ceding control over many important components to large corporate providers, such as Amazon or Rackspace. While "the cloud" remains an option, sysadmins should strive to maintain full openness on their systems, avoid vendor lock-in, and regain control of infrastructure deployment. This talk presents a "full control" look at managing multiple simultaneous FreeBSD deployments around the globe, independently sourced, yet centrally managed. Unlike many common deployments, most of our nodes are physical, rather than virtual, and many are on rented machines where we have little control over the selection of hardware and components. This talk will also cover a number of tools and tricks that were used, obstacles that were overcome, as well as share insights and lessons learned in the process of deploying puppet. Also covers our system for deploying templated jails around the world as part of our CDN and managing them with our Global Server Load Balancer (as discussed at EuroBSDCon 2012). Highlights: * What is puppet? * Deploying puppetmaster for scale (using nginx, not passing large files through ruby) * Managing config files * Managing packages (with portupgrade) * Advanced configuration files with templates * Creating and using custom facts (freebsd specific facts) * Deploying jails with puppet (with ezjail) * Lessons Learned - Delivering large files requires some form of offloading - Templates are where the power is - Puppet is not like scripted deployment, manifests are different (and better) * Where to go from here: - FreeBSD patches for facter, some of our custom facts should be standard - stored configs or puppetdb (needs porting), letting hosts know about each other - using puppet to automatically configure nagios

uk: вхід без авторизації

name: Managing FreeBSD at scale: Reclaiming Control of Large Infrastructure Deployment with Puppet


uk: Комп'ютерні науки

description: Come with me on a journey to learn about the Multipath TCP (MPTCP) protocol and the first publicly released FreeBSD implementation. This talk will examine MPTCP's 'wire' characteristics, the architecture of the modified FreeBSD TCP stack, observations from the development process and results of both performance analysis and empirical research conducted using the stack. Multipath TCP (MPTCP) transparently retrofits multi-pathing capabilities to regular TCP and is a work in progress Internet Draft being developed within the IETF. The Cisco University Research Program funded the Centre for Advanced Internet Architectures to develop an interoperable implementation of MPTCP for FreeBSD as part of a research project to study mixing loss-based and delay-based congestion control in a multipath context. As a researcher on the funded project and lead author of the FreeBSD MPTCP implementation, I've data and insights to share with you about the process of going from stock FreeBSD and an IETF Draft to an interoperable MPTCP implementation that is being used in ongoing research programmes.

uk: вхід без авторизації

name: Multipath TCP for FreeBSD: An overview of the protocol, stack architecture & performance analysis