1 /******************************************************************************* 2 3 Classes to write statistics to be used in graphite. 4 5 Applications that want to log statistics usually make use of the `StatsExt` 6 extension (most likely by deriving from `DaemonApp`), 7 which provides a `StatsLog` instance which is automatically configured from 8 the application's config.ini. 9 10 StatsLog provides methods to: 11 1. Build up a stats line by writing sets of values (specified by the 12 fields of one or more user-specified structs). 13 2. Flush the stats line to the output. 14 15 Currently, `StatsLog` writes to a file (called `stats.log`), which is then 16 parsed by a script that will feed the data to a Collectd socket. 17 Every server's Collectd daemon will then report to a master Collectd server 18 which aggregates the data. 19 As our number of stats is growing and the write rate is increasing, we're 20 planning to expose a way to directly write to the Collectd socket. 21 As a result, the current API of `StatsLog` is intentionally designed 22 to comply to the limitations of Collectd. See the documentation of 23 `StatsLog` for more details 24 25 Refer to the class' description for information about their actual usage. 26 27 Copyright: 28 Copyright (c) 2009-2016 dunnhumby Germany GmbH. 29 All rights reserved. 30 31 License: 32 Boost Software License Version 1.0. See LICENSE_BOOST.txt for details. 33 Alternatively, this file may be distributed under the terms of the Tango 34 3-Clause BSD License (see LICENSE_BSD.txt for details). 35 36 *******************************************************************************/ 37 38 module ocean.util.log.Stats; 39 40 41 42 import ocean.meta.types.Qualifiers; 43 import ocean.core.Verify; 44 import ocean.core.Verify; 45 import ocean.core.Enforce; 46 import ocean.core.ExceptionDefinitions; 47 import ocean.meta.codegen.Identifier /* : fieldIdentifier */; 48 import ocean.core.TypeConvert; 49 import ocean.io.select.EpollSelectDispatcher; 50 import ocean.io.select.client.TimerEvent; 51 import ocean.net.collectd.Collectd; 52 import core.stdc.time : time_t; 53 import ocean.sys.ErrnoException; 54 import ocean.text.convert.Formatter; 55 import ocean.util.container.AppendBuffer; 56 import ocean.util.log.layout.LayoutStatsLog; 57 import ocean.util.log.Appender; 58 import ocean.util.log.Logger; 59 60 61 version (unittest) 62 { 63 import ocean.util.app.DaemonApp; 64 } 65 66 /******************************************************************************* 67 68 Transmit the values of an aggregate to be used within graphite. 69 70 This class has 2 methods which can be used: `add` and `addObject`. 71 72 `add` is meant for application statistics, i.e. amount of memory used, 73 number of channels alive, number of connections open, largest record 74 processed... 75 76 `addObject` logs an instance of an object which belongs to a category. 77 This method should be used when you have a set of standard metrics which 78 you want to log for multiple instances of a type of object. 79 For example, you may want to log standard stats for each channel in 80 a storage engine, for each campaign of an advertiser, 81 for each source of input records, etc. 82 83 See the methods description for more informations. 84 85 Note: 86 StatsLog formerly had the ability to write single value instead of an 87 aggregate. It was removed as it goes against Collectd's design, where 88 data sent to the socket are sent aggregated by 'types', where a type is 89 a collection of related metrics (akin to a `struct`), so single values 90 are not permitted. 91 In addition, it's not possible to incrementally build an aggregate either, 92 as we need the aggregate's complete definition : if we send incomplete/too 93 much data to Collectd, it just rejects the whole aggregate, and data is sent 94 without field names, as Collectd relies on its type definition for that 95 piece of information. Having the wrong order would mean some metrics are 96 logged as other metrics, a bug that might not be easily identifiable. 97 This was leaving too much room for error which were not easily identifiable. 98 99 Examples: 100 See the unittest following this class for an example application 101 102 *******************************************************************************/ 103 104 public class StatsLog 105 { 106 import ocean.util.log.AppendFile; 107 108 /*************************************************************************** 109 110 Stats log config class 111 112 The field `hostname`, `app_name`, `app_instance` and `default_type` 113 are values used by the Collectd integration of `StatsLog`. 114 115 Collectd identify resources using an identifier, that has the 116 following form: 'hostname/plugin-pinstance/type-tinstance'. 117 Every resource written by a process MUST have the same 'hostname', 118 'plugin' and 'pinstance' values. 119 `hostname` value is not limited or checked in any way. 120 Other identifier shall only include alphanum ([a-z] [A-Z] [0-9]), 121 underscores ('_') and dots ('.'). 122 Instance parts can also include dashes ('-'). 123 124 ***************************************************************************/ 125 126 public static class Config 127 { 128 public istring file_name; 129 130 131 /*********************************************************************** 132 133 Path to the collectd socket 134 135 It is null by default. When set through the constructor (usual value 136 is provided through `default_collectd_socket`, `StatsLog` will 137 write to the Collectd socket. 138 139 When this is set, it is required that `app_name` and 140 `app_instance` be set. 141 142 ***********************************************************************/ 143 144 public istring socket_path; 145 146 147 /*********************************************************************** 148 149 'hostname' to use when logging using 'add' 150 151 This is the 'hostname' part of the identifier. 152 If not set, gethostname (2) will be called. 153 See this class' documentation for further details. 154 155 ***********************************************************************/ 156 157 public istring hostname; 158 159 160 /*********************************************************************** 161 162 Collectd 'plugin' name to used 163 164 This is the 'plugin' part of the identifier, and should be set 165 to your application's name. It should hardly ever change. 166 167 By default, the name provided to the application framework will be 168 used. If the application framework isn't used, or the name needs 169 to be overridden, set this value to a non-empty string. 170 171 See this class' documentation for further details. 172 173 ***********************************************************************/ 174 175 public istring app_name; 176 177 178 /*********************************************************************** 179 180 Collectd 'plugin instance' name to used 181 182 This is the 'pinstance' part of the identifier, and should be set 183 to your application's "instance". This can be an id (1, 2, 3...) 184 or a more complicated string, like the ranges over which your app 185 operate ("0x00000000_0x0FFFFFFF"). Change to this value should be 186 rare, if any. 187 The duo of 'plugin' and 'pinstance' should uniquely identify 188 a process (for the same `hostname`). 189 190 See this class' documentation for further details. 191 192 ***********************************************************************/ 193 194 public istring app_instance; 195 196 197 /*********************************************************************** 198 199 Default 'type' to use when logging using 'add' 200 201 This is the 'type' part of the identifier. Usually it is provided 202 as a string template argument to `addObject`, but for convenience, 203 `add` provide a default logging channel. If this argument is not 204 supplied, `collectd_name ~ "_stats"` will be used. 205 206 See this class' documentation for further details. 207 208 ***********************************************************************/ 209 210 public istring default_type; 211 212 213 /*********************************************************************** 214 215 Frequency at which stats are collected 216 217 This metric is expressed in seconds, and should rarely needs to be 218 modified. Defaults to 30s. 219 220 ***********************************************************************/ 221 222 public ulong interval; 223 224 225 /*********************************************************************** 226 227 Constructor 228 229 Emulates struct's default constructor by providing a default value 230 for all parameters. 231 232 ***********************************************************************/ 233 234 public this ( istring file_name = default_file_name, 235 istring socket_path = null, 236 istring hostname = null, 237 istring app_name = null, 238 istring app_instance = null, 239 istring default_type = null, 240 ulong interval = 30) 241 242 { 243 this.file_name = file_name; 244 245 // Collectd settings 246 this.socket_path = socket_path; 247 this.hostname = hostname; 248 this.app_name = app_name; 249 this.app_instance = app_instance; 250 this.default_type = default_type; 251 this.interval = interval; 252 } 253 } 254 255 256 /*************************************************************************** 257 258 Stats log default settings (used in ctor) 259 260 ***************************************************************************/ 261 262 public static immutable time_t default_period = 30; // 30 seconds 263 public static immutable istring default_file_name = "log/stats.log"; 264 265 266 /*************************************************************************** 267 268 Logger instance via which error messages can be emitted 269 270 ***************************************************************************/ 271 272 private Logger error_log; 273 274 275 /*************************************************************************** 276 277 Logger instance via which stats should be output 278 279 ***************************************************************************/ 280 281 protected Logger logger; 282 283 284 /*************************************************************************** 285 286 Message formatter 287 288 ***************************************************************************/ 289 290 protected AppendBuffer!(char) buffer; 291 292 293 /*************************************************************************** 294 295 Whether to add a separator or not 296 297 ***************************************************************************/ 298 299 private bool add_separator = false; 300 301 302 /*************************************************************************** 303 304 Constructor. Creates the stats log using the AppendSysLog appender. 305 306 Params: 307 config = instance of the config class 308 name = name of the logger, should be set to a different string 309 when using more than two StatLogs 310 311 ***************************************************************************/ 312 313 public this ( Config config, istring name = "Stats" ) 314 { 315 Appender newAppender ( istring file, Appender.Layout layout ) 316 { 317 return new AppendFile(file, layout); 318 } 319 320 this(config, &newAppender, name); 321 } 322 323 324 /*************************************************************************** 325 326 Constructor. Creates the stats log using the appender returned by the 327 provided delegate. 328 329 Params: 330 config = instance of the config class 331 new_appender = delegate which returns appender to use for stats log 332 name = name of the logger, should be set to a different string 333 when using more than two StatLogs 334 335 ***************************************************************************/ 336 337 public this ( Config config, 338 scope Appender delegate ( istring file, Appender.Layout layout ) new_appender, 339 istring name = "Stats" ) 340 { 341 if (config.socket_path.length) 342 { 343 verify(config.hostname.length != 0); 344 verify(config.app_name.length != 0); 345 verify(config.default_type.length != 0); 346 } 347 348 // logger via which error messages can be emitted 349 this.error_log = Log.lookup("ocean.util.log.Stats.StatsLog"); 350 351 // logger to which stats should be written 352 this.logger = Log.lookup(name); 353 this.logger.clear(); 354 this.logger.additive(false); 355 356 this.logger.add(new_appender(config.file_name, new LayoutStatsLog)); 357 358 // Explicitly set the logger to output all levels, to avoid the situation 359 // where the root logger is configured to not output level 'info'. 360 this.logger.level = this.logger.Level.Trace; 361 362 this.buffer = new AppendBuffer!(char); 363 364 if (config.socket_path.length) 365 { 366 // Will throw if it can't connect to the socket 367 this.collectd = new Collectd(config.socket_path); 368 369 this.identifier.host = config.hostname; 370 this.identifier.plugin = config.app_name; 371 this.identifier.plugin_instance = config.app_instance; 372 this.identifier.type = config.default_type; 373 this.options.interval = config.interval; 374 } 375 } 376 377 378 /*************************************************************************** 379 380 Adds the values of the given aggregate to the stats log. Each member 381 of the aggregate will be output as <member name>:<member value>. 382 383 Params: 384 values = aggregate containing values to write to the log. 385 386 ***************************************************************************/ 387 388 public typeof(this) add ( T ) ( T values ) 389 { 390 static assert (is(T == struct) || is(T == class), 391 "Parameter to add must be a struct or a class"); 392 this.format!(null)(values, istring.init); 393 if (this.collectd !is null) 394 this.sendToCollectd!(null)(values, istring.init); 395 return this; 396 } 397 398 399 /*************************************************************************** 400 401 Adds the values of the given aggregate to the stats log. Each member of 402 the aggregate will be output as 403 <category>/<instance>/<member name>:<member value>. 404 405 Params: 406 category = The name of the category this object belongs to. 407 408 instance = Name of the object to add. 409 values = aggregate containing values to write to the log. 410 411 ***************************************************************************/ 412 413 public typeof(this) addObject (istring category, T) 414 (cstring instance, T values) 415 { 416 static assert (is(T == struct) || is(T == class), 417 "Parameter to add must be a struct or a class"); 418 static assert(category.length, 419 "Template parameter 'category' should not be null"); 420 verify (instance.length != 0, "Object name should not be null"); 421 422 this.format!(category)(values, instance); 423 if (this.collectd !is null) 424 this.sendToCollectd!(category)(values, instance); 425 return this; 426 } 427 428 429 /*************************************************************************** 430 431 Flush everything to file and prepare for the next iteration 432 433 ***************************************************************************/ 434 435 public void flush ( ) 436 { 437 this.logger.info(this.buffer[]); 438 this.add_separator = false; 439 this.buffer.clear(); 440 } 441 442 443 /*************************************************************************** 444 445 Writes the values from the provided aggregate to the format_buffer 446 member. 447 448 Each member of the aggregate is output as either: 449 <category name>/<object name>/<member name>:<member value> 450 if a category is provided, or as: 451 <member name>:<member value> 452 if no category is provided. 453 It's a runtime error to provide a category but no instance name, or the 454 other way around. 455 456 Note: When the aggregate is a class, the members of the super class 457 are not iterated over. 458 459 Params: 460 category = the type or category of the object, such as 'channels', 461 'users'... May be null (see the 'instance' parameter). 462 T = the type of the aggregate containing the fields to log 463 464 values = aggregate containing values to write to the log. Passed as 465 ref purely to avoid making a copy -- the aggregate is not 466 modified. 467 instance = the name of the instance of the category, or null if 468 none. For example, if the category is 'companies', then the name 469 of an instance may be "google". This value should be null if 470 category is null, and non-null otherwise. 471 472 ***************************************************************************/ 473 474 private void format ( istring category, T ) ( ref T values, cstring instance ) 475 { 476 scope sink = (cstring v) { this.buffer ~= v; }; 477 foreach ( i, value; values.tupleof ) 478 { 479 auto value_name = .identifier!(T.tupleof[i]); 480 481 static if (is(typeof(value) : long)) 482 long fmtd_value = value; 483 else static if (is(typeof(value) : double)) 484 double fmtd_value = value; 485 else 486 { 487 pragma(msg, "[", __FILE__, ":", __LINE__, "] '", T.stringof, 488 "' should only contain integer or floating point members"); 489 auto fmtd_value = value; 490 } 491 492 // stringof results in something like "values.somename", we want 493 // only "somename" 494 if (this.add_separator) 495 { 496 this.buffer ~= ' '; 497 } 498 499 static if (category.length) 500 { 501 verify(instance.length != 0); 502 sformat(sink, "{}/{}/{}:{}", 503 category, instance, value_name, fmtd_value); 504 } 505 else 506 { 507 verify(!instance.length); 508 sformat(sink, "{}:{}", value_name, fmtd_value); 509 } 510 511 this.add_separator = true; 512 } 513 } 514 515 516 /*************************************************************************** 517 518 Send the content of the struct to Collectd 519 520 This will format and send data to the Collectd daemon. 521 If any error happen, they will be reported by a log message but won't 522 be propagated up, so applications that have trouble logging won't 523 fail. 524 525 Note: 526 As with `format`, when the aggregate is a class, the members of 527 the super class are not iterated over. 528 529 Params: 530 category = the type or category of the object, such as 'channels', 531 'users'... May be null (see the 'instance' parameter). 532 T = the type of the aggregate containing the fields to log 533 534 values = aggregate containing values to write to the log. Passed as 535 ref purely to avoid making a copy -- the aggregate is not 536 modified. 537 instance = the name of the instance of the category, or null if 538 none. For example, if the category is 'companies', then the name 539 of an instance may be "google". This value should be null if 540 category is null, and non-null otherwise. 541 542 ***************************************************************************/ 543 544 private void sendToCollectd (istring category, T) (ref T values, 545 cstring instance) 546 { 547 verify(this.collectd !is null); 548 static if (!category.length) 549 verify(!instance.length); 550 551 // It's a value type (struct), do a copy 552 auto id = this.identifier; 553 static if (category.length) 554 { 555 id.type = category; 556 id.type_instance = instance; 557 } 558 559 // putval returns null on success, a failure reason else 560 try 561 this.collectd.putval!(T)(id, values, this.options); 562 catch (CollectdException e) 563 this.error_log.error("Sending stats to Collectd failed: {}", e); 564 catch (ErrnoException e) 565 this.error_log.error("I/O error while sending stats: {}", e); 566 } 567 568 569 /*************************************************************************** 570 571 Collectd instance 572 573 ***************************************************************************/ 574 575 protected Collectd collectd; 576 577 578 /*************************************************************************** 579 580 Default identifier when doing `add`. 581 582 ***************************************************************************/ 583 584 protected Identifier identifier; 585 586 587 /*************************************************************************** 588 589 Default set options to send 590 591 Currently it's only `interval=30`. 592 593 ***************************************************************************/ 594 595 protected Collectd.PutvalOptions options; 596 } 597 598 /// Usage example for StatsLog in a simple application 599 unittest 600 { 601 class MyStatsLogApp : DaemonApp 602 { 603 private static struct Stats 604 { 605 double awesomeness; 606 double bytes_written; 607 double bytes_received; 608 } 609 610 private static struct Channel 611 { 612 double profiles_in; 613 double profiles_out; 614 } 615 616 public this () 617 { 618 super("Test", null, null); 619 } 620 621 protected override int run (Arguments args, ConfigParser config) 622 { 623 auto epoll = new EpollSelectDispatcher; 624 this.startEventHandling(epoll); 625 return 0; 626 } 627 628 protected override void onStatsTimer ( ) 629 { 630 // Do some heavy-duty processing ... 631 Stats app_stats1 = { 42_000_000, 10_000_000, 1_000_000 }; 632 Stats app_stats2 = { 42_000_000, 1_000_000, 10_000_000 }; 633 this.stats_ext.stats_log.add(app_stats1); 634 635 // A given struct should be `add`ed once and only once, unless 636 // you flush in between 637 this.stats_ext.stats_log.flush(); 638 this.stats_ext.stats_log.add(app_stats2); 639 640 // Though if you use `addObject`, it's okay as long as the instance 641 // name is different 642 Channel disney = { 100_000, 100_000 }; 643 Channel discovery = { 10_000, 10_000 }; 644 645 // For the same struct type, you probably want the 646 // same category name. It's not a requirement but there are 647 // no known use case where you want it to differ. 648 this.stats_ext.stats_log.addObject!("channel")("disney", disney); 649 this.stats_ext.stats_log.addObject!("channel")("discovery", discovery); 650 } 651 } 652 }