public interface CacheSynchronization

Automatic Cache Synchronization

Cache synchronization is the mechanism used by Smart GWT to ensure that updates made by dataBoundComponents or programmatically on dataSources directly, are reflected in the client-side caches of other dataBoundComponents. Automatic cache sync means that components update themselves to reflect data changes - so if you update a record in a form, and the same information is currently visible in a grid, the grid data will change to reflect the updates automatically.

The Smart GWT server is designed to retrieve cache sync data in the most optimal way, taking into account the capabilities of the target data store or data service and the context of the request, such as whether it came from a browser or was programmatically initiated on the server. The cacheSyncStrategy setting can be used to manually control how cache sync is performed server-side, whenever that's necessary.

CacheSyncStrategy

Cache sync configuration revolves around the CacheSyncStrategy type, and to a lesser extent the CacheSyncTiming. These settings allow you to define a default cache sync approach globally, overridable per-DataSource, per-Operation or per-request. See the CacheSyncStrategy documentation for details.

Note that DataSource-level, operation-level and request-level cacheSyncStrategy settings are honored in all cases (though see below for a caveat on that), but global settings are defaults only. Smart GWT will override the default if it detects that it might lead to problems in a given case.

For example, if a DataSource declares a field that specifies the autoGenerated flag, it is saying that field is not a sequence but its value is nevertheless generated by the database or ORM platform, so we would not expect a value for that field in the request values. So if the global default cacheSyncStrategy is "requestValuesPlusSequences", that is going to lead to Smart GWT returning an incomplete cache sync record, which might well look to your users like a broken application.

Smart GWT will detect these cases where we would have possibly incomplete cache sync data, and automatically change the strategy to "refetch". This is done when we have a field that is part of the expected outputs (see DSRequest.outputs and OperationBinding.outputs) and specifies any of the following properties (because they all mean that the field's value is server-generated in some way):

• autoGenerated
• customSQL
• customSelectExpression

Similarly, Smart GWT will override the global cacheSyncTiming setting according to the requirements of a given request. For example, if the global default is "lazy" and there is no DataSource-level, operation-level or request-level timing setting, Smart GWT will override the timing to "immediate" for any client-originated request, amongst other things. See the CacheSyncTiming documentation for further details.

If you do not want these intelligent fallback behaviors for a given DataSource, operation or request, set a cacheSyncStrategy or cacheSyncTiming on the DataSource, operation or request. This will always be honored for cacheSyncStrategy, even when the system knows that it is going to lead to incomplete cache-sync data. However, even an explicit cacheSyncTiming setting at the DataSource, operation or request level will be ignored in certain circumstances, where deferring the cache sync operation could break framework functionality. An example of this is auditing: if your dataSource is configured for automatic auditing, the framework categorically does need the cache sync data in order to write an audit record, and when deferred cache sync is in force, we have no guarantee that cache sync will run at all. So we ignore attempts to configure it for deferred cache sync.

CacheSyncStrategy support with different DataSource types

Different DataSource types may support only a subset of cacheSyncStrategy options, since some strategies are specific to the capabilities of specific data stores or remote services. The following section describes how CacheSyncStrategy applies to various built-in dataSource types

SQLDataSource

SQL databases accessed via JDBC do not return the record-as-saved, so extra work is required to retrieve database-generated fields, such as sequences (though sequences are a special case, see below)

With JDBC 3.0+ drivers and a sequenceMode of jdbcDriver, we can retrieve generated sequence values without requiring an explicit, separate SQL query to retrieve the generated keys. With the default CacheSyncStrategy of requestValuesPlusSequences, SQLDataSource uses this JDBC approach to avoid a separate SQL query where possible - see the "requestValuesPlusSequences" section of the CacheSyncStrategy documentation for details. With a sequenceMode of native, we will issue a separate SQL query to retrieve the generated keys, even if you are using "requestvaluesPlusSequences". However, it is not a full refetch, it is just a special native query, specific to the database in use, to retrieve just the generated sequence values, so it is still likely to have a performance advantage over using "refetch" - in the refetch case, we perform an additional full fetch, using the retrieved keys as the criteria.

Note that sequences (or identity columns, or auto-increment columns - different databases use different mechanisms and terminology for the same concept) are not the only kind of database-generated value. As well as the three DataSource field settings, mentioned above, that can explicitly denote a generated value - autoGenerated, customSQL and customSelectExpression - database-generated values can come from database-declared default values, UDFs and stored procedures, and triggers. In addition, application code can modify data returned from the database in arbitrary ways, via logic in a DMI, server script or custom DataSource implementation. So there are cases where "refetch" is necessary, and for some applications it may be that most or even all cases require it (if you make extensive use of triggers, or enhance many database responses in Java code, for example).

Sequence field object type considerations

SQL databases support different underlying column types for sequence fields. Typically, you use either INTEGER or BIGINT, depending on the maximum number of rows you anticipate for a given table. Unfortunately, the different database products are not always consistent in the way they convert values from the different underlying SQL types into Java objects. For example, MySQL returns values from SQL INTEGER columns as instances of java.lang.Integer in ordinary fetch ResultSets, but as instances of java.lang.Long from the getGeneratedKeys() API.

This is a problem if you have existing Java code that is expecting an instance of Integer; if you switch to the more efficient "requestValuesPlusSequences" cacheSyncStrategy, your code will be sent an instance of Long instead. Note, this is only a problem for existing server-side Java code; client code is not affected because the Smart GWT framework already coerces all integer-type values to type Long for serialization to the client.

The Smart GWT framework offers a number of configurable mitigations for this problem:

• We maintain a table of known Java types returned by each database product for "small" sequence columns (32-bit, SQL type INT or INTEGER) and "large" sequence columns (64-bit, SQL type BIGINT). We also maintain a flag per database product indicating whether "small" or "large" sequence columns should be used (as mentioned in the bullet-point below, this applies to integer key fields generally, not just sequences). You can override this flag in your server.properties file, per database product or per dbName (the latter is more specific and wins if there is a conflict). For example:

       # Change all Postgres db connections to use small keys
       sql.postgresql.defaultKeySize: small
       # Change the db with the specific dbName "SalesDB" to use large keys
       sql.SalesDB.defaultKeySize: large
    

• If you need Java types other than the defaults described above (java.lang.Integer for "small" sequences, java.lang.Long for "large" sequences), you can declare entries in server.properties to override them, per database product or per dbName, like this:

       # For Postgres, large key values should be returned as instances of BigInteger
       sql.postgresql.javaTypeForLargeKey: java.math.BigInteger
       # and small key values should be returned as instances of Short
       sql.postgresql.javaTypeForSmallKey: java.lang.Short
       # But small key values should be returned as type BigDecimal for AdminDB
       sql.AdminDB.javaTypeForSmallKey: java.math.BigDecimal
    

  This is useful for new projects that may later switch database vendors, or for projects that use multiple database products in tandem, because it can be used to enforce cross-database type consistency. We would recommend using java.lang.Integer and java.lang.Long for small and large sequences, respectively, since these cover exactly the same range of valid values as the corresponding SQL types, INTEGER and BIGINT


• The server.properties flag sql.transformGeneratedKeys, on by default, causes the framework to transform the object returned by the getGeneratedKeys() API in accordance with the above. So, if "SalesDB" and "AdminDB" are both PostgreSQL databases, the settings above would mean:
  • Adding a record to a DataSource with dbName="SalesDB" would return an instance of java.math.BigInteger for the sequence field
  • Adding a record to a DataSource with dbName="AdminDB" would return an instance of java.math.BigDecimal for the sequence field (because of the custom javaTypeForSmallSequence declared for that dbName)


• Note that the above rules are also applied by Smart GWT's table-creation logic when you use the Admin Console to import SQL DataSources. Additionally, we use these rules to decide on a column type for any integer field that is marked as primaryKey or foreignKey, to ensure that the same SQL type is used at both ends of the relation (because some databases require this)


• If sql.transformGeneratedKeys is explicitly set false, no transformation takes place; your code is directly returned whatever object type the JDBC dirver returned


• If you have an existing project where a mix of large and small sequence types is already in place, the above approaches to provide consistency will not help. In this case, the server.properties flag "sql.transformGeneratedKeysToFetchType" should be considered. This flag is off by default; when switched on, it causes the framework to cast the generated value of a sequence field, as returned by the JDBC driver or native query, to the same Java type that a regular fetch of that field would have returned. This is accomplished by running a basic fetch of a single row the first time the information is required, determining the underlying SQL type from the ResultSetMetadata, and then caching that information for future use. The performance impact of this is minor, since it involves a single additional fetch per DataSource instance for the lifetime of the JVM. Note, this flag has no effect if sql.transformGeneratedKeys is false


• sql.transformGeneratedKeysToFetchType is only intended for existing projects that have a lot of server-side code that interacts with the Smart GWT Server and may be affected by cacheSyncStrategy "requestValuesPlusSequences" returning a different Java object type for sequence fields than your existing code is expecting. For any new project we would recommend using the other options outlined above to either match your database's return type for fetches, or to force consistency to types you specify


• Finally, the Java type to use for a specific field can be set using the field's javaClass property. Alternatively, Smart GWT will automatically coerce values to match declared types if you are using Javabeans rather than Maps as your data model - see beanClassName. Both of these approaches take precedence over the mechanisms described above

HibernateDataSource and JPADataSource

HibernateDataSource and JPADataSource only support refetch. These two implementations integrate with the underlying ORM system at the level of the ORM's API, allowing it to handle the details of database interaction. With these two DataSource types, we are simply working with "persistent objects" - how the ORM manages things like changes made by the database during update queries, or sequence values in add operations, is the ORM's business.

For this reason, HibernateDataSource and JPADataSource install a special CacheSyncStrategy implementation under the refetch name, that just does nothing, leaving the response data returned by the update operation unchanged.

Custom/Generic DataSources

In addition to the built-in DataSource types listed above, you can of course write your own custom dataSource implementations. These custom DataSources will participate in cache sync like any other:

• You can specify a cacheSyncStrategy on the DataSource, operationBinding or dsRequest
• The default strategy for a custom dataSource is "responseValues", because that was the prevailing behavior for custom DataSources before cacheSyncStrategy was introduced
• If you want to use "refetch" (ie, you override the default in your server.properties, or set the strategy explicitly on your DataSource or operation binding), you must implement a fetch operation, and if your dataSource has fields of type "sequence", your fetch mechanism must be able to resolve the values of such fields

Note that "refetch" with a custom dataSource is done lazily, and it is not done at all if nothing asks for the response data. This is because the integration with cache sync happens when the server-side DSResponse's getData() method is called. This happens automatically and will work perfectly for most use cases. If, however, you have some unusual requirement which means you need "refetch" to cause an immediate cache sync fetch like it does with the built-in dataSources, you can do what they do: invoke the CacheSyncStrategy manually from your execution flow, like this:

     CacheSyncStrategy strategy = dsRequest.getCacheSyncStrategy();
     if (strategy.shouldRunCacheSync(dsRequest)) {
         // Apply the cache sync data to the dsResponse, first fetching it if necessary
         strategy.applyCacheSyncStrategy(dsRequest, dsResponse)
     }
  

canSyncCache, cacheSyncOperation and useForCacheSync

These three long-standing operationBinding flags interact with the above-documented behavior of CacheSyncStrategy as follows:

• If canSyncCache is false, no cache sync logic will run at all
• If a useForCacheSync operation is in force, or the update operation specifies a cacheSyncOperation, that operation will be run if we are refetching the updated record - ie, if the cacheSyncStrategy is refetch. Depending on the cacheSyncTiming in force, the operation execution may be deferred, and may not run at all. If we are not refetching the updated record, these two flags have no effect

Adding multiple records

By default, cache sync is switched off for multi-insert requests (ie, add requests with more than one valueSet; cache sync is always off for allowMultiUpdate operations, unless you explicitly specify a cacheSyncStrategy on the request or operationBinding). You can change this default behavior for multi-inserts by changing server.properties flag "default.multi.update.cache.sync.strategy" to "sync""; this will cause the system to use the same cacheSyncStrategy it would use for a regular single-record request on that DataSource. Note that in this case, the default strategy can be auto-overridden by the framework just like a normal single-update strategy.

You can also just set a specific cacheSyncStrategy on the DataSource, operation or DSRequest, just like with a regular single-record request, and again, these specific settings are not auto-overridden except in cases where they could potentially cause feature breakage, as described above.